ROBUST UNLEARNABLE EXAMPLES: PROTECTING DATA PRIVACY AGAINST ADVERSARIAL LEARNING
Year: 2021

Abstract:
The tremendous amount of accessible data in cyberspace face the risk of being unauthorized used for training deep learning models. To address this concern, methods are proposed to make data unlearnable for deep learning models by adding a type of error-minimizing noise. However, such conferred unlearnability is found fragile to adversarial training. In this paper, we design new methods to generate robust unlearnable examples that are protected from adversarial training. We first find that the vanilla error-minimizing noise, which suppresses the informative knowledge of data via minimizing the corresponding training loss, could not effectively minimize the adversarial training loss. This explains the vulnerability of error-minimizing noise in adversarial training. Based on the observation, robust error-minimizing noise is then introduced to reduce the adversarial training loss. Experiments show that the unlearnability brought by robust error-minimizing noise can effectively protect data from adversarial training in various scenarios.

heading: 1 INTRODUCTION
text: Recent advances in deep learning largely rely on various large-scale datasets, which are mainly built upon public data collected from various online resources such as Flickr, Google Street View, and different search engines (Lin et al., 2014; Torralba et al., 2008; Netzer et al., 2011). However, the process of data collection might be unauthorized, leading to the risk of privacy leakage and personal data misuse. For example, recent work demonstrates that individual information such as name or email address could be leaked from a pre-trained GPT-2 model (Carlini et al., 2020). Another investigation reveals that a company has been continuously collecting facial images from the internet for training commercial facial recognition models (Hill, 2020).
To protect the privacy and prevent the unauthorized use of the released data, some studies suggest poisoning the clean data with imperceptible noise so that the performances of models trained on the modified data be significantly downgraded (Fowl et al., 2021b;a; Yuan & Wu, 2021; Huang et al., 2021). Our work mainly focuses on the work by Huang et al. (2021). Specifically, Huang et al. (2021) introduce unlearnable examples that are crafted from clean data via adding a type of imperceptible error-minimizing noise and show impressive data-protection ability. The error-minimizing noise is designed based on the intuition that an example of higher training loss may contain more knowledge to be learned. Thereby, the noise protects data from being learned via minimizing the corresponding loss to suppress the informative knowledge of data as much as possible.
However, such an unlearnability conferred by error-minimizing noise is found vulnerable to adversarial training (Huang et al., 2021; Tao et al., 2021), a standard approach towards training robust deep learning models (Madry et al., 2018). In adversarial training, a model trained on the unlearnable data can achieve the same performance as that trained on the clean data. Similar problems are also found in other poisoning-based data protection methods, e.g., the targeted adversarial poisoning method (Fowl et al., 2021b) (see Fig. 1). These findings bring great challenges to poisoning-based data privacy protection, and further raises the following question:
Can we still make data unlearnable by adding imperceptible noise even under the adversarial training?
In this paper, we give an affirmative answer that imperceptible noise can indeed stop deep learning models from learning high-level knowledge in an adversarial manner. We first analyze the reason
why the unlearnability conferred by error-minimizing noise fails under adversarial training. Adversarial training improves the robustness of models via training them on adversarial examples. However, adversarial examples usually correspond to higher loss, which according to Huang et al. (2021) may contain more knowledge to be learned. Though the error-minimizing noise is designed to minimize the training loss, adversarial examples crafted from unlearnable data could still significantly increase the loss scale (see Fig. 2). Therefore, adversarial training can break the protection brought by error-minimizing noise and still learn knowledge from these data.
Based on the aforementioned analysis, we further deduce that the invalidation of unlearnable examples in adversarial training might be attributed to the error-minimizing noise could not effectively reduce the adversarial training loss. To this end, a new type of noise, named robust error-minimizing noise, is designed to protect data privacy from adversarial learners via minimizing the adversarial training loss. Inspired by the strategy of adversarial training and the generation of error-minimizing noise, a min-min-max optimization is designed to train the robust error-minimizing noise generator. Examples crafted by adding robust error-minimizing noise are called robust unlearnable examples.
In summary, our work has three main contributions: (1) We present robust error-minimizing noise, which is the first and currently the only method that can prevent data from being learned in adversarial training to the best of our knowledge. (2) We propose to solve a min-min-max optimization problem to effectively generate robust error-minimizing noise. (3) We empirically verify the effectiveness of robust error-minimizing noise under adversarial training.

heading: 2 RELATED WORKS
text: Adversarial attacks. Adversarial examples are carefully crafted modifications of normal examples that can fool machine learning models to behave in unexpected ways. It has been found that adversarial examples widely exist and machine learning models are vulnerable to adversarial examples (Szegedy et al., 2013; Goodfellow et al., 2015; Carlini & Wagner, 2017). Though minor data transformations can usually eliminate the adversaries of adversarial examples (Lu et al., 2017; Luo et al., 2015), many works have demonstrated that adversarial examples can further be made resistant against these transformations (Evtimov et al., 2017; Eykholt et al., 2018; Wu et al., 2020b; Athalye et al., 2018; Liu et al., 2019; 2020a). This makes adversarial attacks realistic threatens.
To tackle adversarial attacks, adversarial training is proposed to improve the robustness of deep learning models against adversarial examples. A standard adversarial training algorithm aims to solve a minimax problem, which minimizes the loss function while the most adversarial examples present (Madry et al., 2018). Other advances include TRADE (Zhang et al., 2019), FAT (Zhang et al., 2020), GAIRAT (Zhang et al., 2021), RLFAT (Song et al., 2020), Jiang et al. (2021), Stutz et al. (2020), Singla & Feizi (2020), Croce & Hein (2020), Wu et al. (2020a), and Wu et al. (2021). Tang et al. (2021) propose an adversarial robustness benchmark regarding architecture design and training techniques. Besides, there are also some works that attempt to interpret how machine learning models gain robustness through adversarial training (Ilyas et al., 2019; Zhang & Zhu, 2019).
Poisoning attacks. Poisoning attacks aim to manipulate the performance of a machine learning model via injecting malicious poisoned examples into its training data set (Biggio et al., 2012; Koh & Liang, 2017; Shafahi et al., 2018; Liu et al., 2020b; Jagielski et al., 2018; Yang et al., 2017; Steinhardt et al., 2017). The generated poisoned examples usually appear different from the clear examples (Biggio et al., 2012; Yang et al., 2017; Steinhardt et al., 2017). However, recent approaches show that poisoned examples can also be crafted imperceptible to their corresponding origins (Koh & Liang, 2017; Shafahi et al., 2018). A special case of poisoning attacks is the backdoor attack. A model trained on backdoored data may perform well on normal data but wrongly behave on data that contain trigger patterns (Chen et al., 2017; Nguyen & Tran, 2020; 2021; Li et al., 2021; Weng et al., 2020). Comparing to other poisoning attacks, backdoor attack is more covert and thus can be used to implement more harmful attacks, such as supply chain attacks (Gu et al., 2017).
Recent works start to employ data poisoning to protect data privacy and prevent unauthorized model training. Huang et al. (2021) propose to modify data with imperceptible error-minimizing noise to stop deep learning model learning knowledge from the modified data. Yuan & Wu (2021) adopt a Gaussian Process that is designed based on Neural Tangent Kernels to model the mean of a class of wide DNNs, and generate protective noise upon it. This method can enhance the transferability of the generated noise across different models. Fowl et al. (2021a) and Fowl et al. (2021b) employ gradient alignment (Geiping et al., 2020; 2021) and PGD method (Madry et al., 2018) to generate adversarial examples respectively, and then use the generated data as poisoned data to downgrade the performance of models trained on them. However, Tao et al. (2021) find that existing poisoningbased protection can be easily broken by adversarial training.

heading: 3 PRELIMINARIES
text: Suppose D = {(x1, y1), · · · , (xn, yn)} is a dataset consists of n samples, where xi ∈ X is the feature of the i-th sample and yi ∈ Y is the corresponding label. A parameterized machine learning model is fθ : X → Y , where θ ∈ Θ is the model parameter. Suppose ` : Y × Y → [0, 1] is a loss function. Then, the empirical risk minimization (ERM) aims to approach the optimal model by solving the following optimization problem,
min θ
1
n n∑ i=1 `(fθ(xi), yi). (1)
Adversarial training (Madry et al., 2018) improves the robustness of models against adversarial attacks via training them on adversarial examples. A standard adversarial training aims to solve the following min-max problem,
min θ
1
n n∑ i=1 max ‖δi‖≤ρa `(fθ(xi + δi), yi), (2)
where ρa > 0 is the adversarial perturbation radius and (xi + δi) is the most adversarial example within the ball sphere centered at xi with radius ρa. Usually, a larger radius ρa implies stronger adversarial robustness of the trained model.
Unlearnable examples (Huang et al., 2021) is a type of data that deep learning models could not effectively learn informative knowledge from them. Models trained on unlearnable examples could only achieve severely low performance. The generation of unlearnable examples contains two steps. Firstly, train an error-minimizing noise generator f ′θ via solving the following optimization problem,
min θ
1
n n∑ i=1 min ‖δi‖≤ρu `(f ′θ(xi + δi), yi), (3)
where ρu is the defensive perturbation radius that forces the generated error-minimizing noise to be imperceptible. Then, an unlearnable example (x′, y) is crafted via adding error-minimizing noise generated by the trained noise generator f ′(θ) to its clean counterpart (x, y), where x′ = x + arg min‖δ‖≤ρu `(f ′ θ(x + δ), y). The intuition behind unlearnable examples is, a smaller loss may imply less knowledge that could be learned from an example. Thus, error-minimizing noise aims to make data unlearnable via reducing the corresponding training loss. It is worth noting that once the unlearnable data is released publicly, the data defender could not modify the data any further.
Projected gradient descent (PGD) (Madry et al., 2018) is a standard approach for solving the inner maximization and minimization problems in Eqs. (2) and (3). It performs iterative projection updates to search for the optimal perturbation as follows,
δ(k) = ∏ ‖δ‖≤ρ [ δ(k−1) + c · α · sign ( ∂ ∂δ l(fθ(x+ δ (k−1)), y) )] ,
where k is the current iteration step (K steps at all), δ(k) is the perturbation found in the k-th iteration, c ∈ {−1, 1} is a factor for controlling the gradient direction, α is the step size, and ∏ ‖δ‖≤ρ means the projection is calculated in the ball sphere {δ : ‖δ‖ ≤ ρ}. The final output perturbation is δ(K). Throughout this paper, the coefficient c is set as 1 when solving maximization problems and −1 when solving minimization problems.

heading: 4 ROBUST UNLEARNABLE EXAMPLES
text: In this section, we first illustrate the difficulty of reducing adversarial training loss with errorminimizing noise, and then introduce robust unlearnable examples to tackle the raised challenge.

heading: 4.1 ERROR-MINIMIZING NOISE IN ADVERSARIAL TRAINING
text: Error-minimizing noise prevents unauthorized model training via reducing the corresponding loss to suppress the learnable knowledge of data. However, the goal of reducing training loss may be at odds with that of adversarial training. Recall Eq. (2), adversarial examples usually correspond to higher training losses than that of the clean data, which according to Huang et al. (2021) may contain more knowledge to be learned. Even if error-minimizing noise presents, crafting adversarial examples from unlearnable data could still significantly increase the training loss.
To better illustrate this phenomenon, we conduct adversarial training on clean data and unlearnable data with different adversarial perturbation radius ρa. The training losses along the training process are collected and plotted in Fig. 2. From the figure, it can be found that error-minimizing noise could not reduce the training loss as effectively as that in ERM training. Furthermore, as the adversarial perturbation radius ρa increases, the adversarial training loss curves on unlearnable data eventually coincide with that on clean data. These observations suggest that adversarial training can effectively recover the knowledge of data protected by error-minimizing noise, which makes the conferred unlearnability invalid.

heading: 4.2 ROBUST ERROR-MINIMIZING NOISE
text: Motivated by the empirical study in Section 4.1, we propose a type of imperceptible noise, named robust error-minimizing noise, to preserve data privacy against adversarial learners. Robust errorminimizing noise is designed to minimize the adversarial training loss during adversarial training,
which intuitively can barrier the knowledge learning process in adversarial training. Examples crafted via adding robust error-minimizing noise are named robust unlearnable examples.
Similar to that in Huang et al. (2021), the generation of robust error-minimizing noise also needs to first train a noise generator f ′θ. Inspired by the adversarial training loss in Eq. (2) and the objective function for training error-minimizing noise generator in Eq. (3), a min-min-max optimization process is designed to train the robust error-minimizing noise generator f ′θ as follows,
min θ
1
n n∑ i=1 min ‖δui ‖≤ρu max ‖δai ‖≤ρa `(f ′θ(xi + δ u i + δ a i ), yi), (4)
where the defensive perturbation radius ρu forces the generated noise to be imperceptible, and the adversarial perturbation radius ρa controls the protection level of the noise against adversarial training. The idea behind Eq. (4) is, we aim to find a “safer” defensive perturbation δu∗i such that the adversarial example crafted from the protected sample point (xi+δu∗i ) would not increase the training loss too much. As a result, the adversarial learner could not extract much knowledge from the data protected by this safer defensive perturbation.
For the choices of the two perturbation radii ρu and ρa, we notice that when ρu ≤ ρa, the following inequality holds for each summation term in Eq. (4),
min ‖δui ‖≤ρu max ‖δai ‖≤ρa
`(f ′θ(xi + δ u i + δ a i ), yi) ≥ `(f ′θ(xi), yi).
The above inequality suggests that when ρu ≤ ρa, the generated defensive noise δui could not suppress any learnable knowledge of data even in ERM training. Thereby, to help the generated noise gain remarkable protection ability, the radius ρu should be set larger than ρa.

heading: 4.3 ENHANCING THE STABILITY OF THE NOISE
text: However, the unlearnability conferred by the noise generated via Eq. (4) is found to be fragile to minor data transformation. For example, standard data augmentation (Shorten & Khoshgoftaar, 2019) can easily make the conferred unlearnability invalid (see Section 5.2). To this end, we adopt the expectation over transformation technique (EOT; Athalye et al. 2018) into the generation of robust error-minimizing noise. EOT is a stability-enhancing technique that was first proposed for adversarial examples (Evtimov et al., 2017; Eykholt et al., 2018; Athalye et al., 2018). Since adversarial examples and robust unlearnable examples are similar to some extent, it is likely that EOT can also help to improve the stability of robust error-minimizing noise.
Suppose T is a given distribution over a set of some transformation functions {t : X → X}. Then, the objective function for training robust error-minimizing noise generator with EOT is adapted from Eq. (4) as follows,
min θ
1
n n∑ i=1 min ‖δui ‖≤ρu Et∼T max ‖δai ‖≤ρa `(f ′θ(t(xi + δ u i ) + δ a i ), yi). (5)
Eq. (5) suggests that, when searching for the defensive perturbation δui , one needs to minimize several adversarial losses of a set of the transformed examples rather than only minimize the adversarial losses of single examples. After finishing training the noise generator via Eq. (5), the robust unlearnable example (x′, y) for a given data point (x, y) is generated as
x′ = x+ arg min ‖δu‖≤ρu Et∼T max ‖δa‖≤ρa `(f ′θ(t(x+ δ u) + δa), y).

heading: 4.4 EFFICIENTLY TRAINING THE NOISE GENERATOR
text: To train the robust error-minimizing noise generator with Eq. (5), we employ PGD to solve the inner minimization and maximization problems. The main challenge of this approach is the gradient calculation of the inner maximization function.
Specifically, the inner maximization function in Eq. (5) usually does not have an analytical solution. As a result, its gradient calculation could not be directly handled by modern Autograd systems such as PyTorch and TensorFlow. Toward this end, we approximate the gradient via first calculating
Algorithm 1 Training robust error-minimizing noise generator with Eq. (5) Input: Training data set D, training iteration M ,
1: PGD parameters ρu, αu and Ku for solving the minimization problem, 2: PGD parameters ρa, αa and Ka for solving the maximization problem, 3: The data transformation distribution T , 4: the number of sampling time J when approximating the gradient of the expectation function.
Output: Robust error-minimizing noise generator f ′θ. 5: Initialize source model parameter θ. 6: for i in 1, · · · ,M do 7: Sample a minibatch (x, y) ∼ D. 8: Initialize δu. 9: for k in 1, · · · ,Ku do 10: for j in 1, · · · , J do 11: Sample a transformation function tj ∼ T . 12: δaj ← PGD(tj(x+ δu), y, f ′θ, ρa, αa,Ka) . Finding adversarial perturbation. 13: end for 14: gk ← 1J ∑J j=1 ∂ ∂δu `(f ′ θ(tj(x+ δ u) + δaj ), y)
15: δu ← ∏ ‖δ‖≤ρu (δ
u − αu · sign(gk)) 16: end for 17: Sample a transformation function t ∼ T . 18: δa ← PGD(t(x+ δu), y, f ′θ, ρa, αa,Ka) 19: Update source model parameter θ based on minibatch (t(x+ δu) + δa, y). 20: end for 21: return f ′θ
the optimal perturbation δai ∗ = arg max‖δai ‖≤ρa `(f ′ θ(t(xi + δ u i ) + δ a i ), yi) with PGD, and then approximating the gradient of the maximization function by ∂∂δui `(f ′ θ(t(xi + δ u i ) + δ a i ∗), yi).
Therefore, when searching for the optimal defensive perturbation δu∗i with PGD, one can follow Athalye et al. (2018) to approximate the gradient of the expectation of transformation via first sampling several transformation functions and then averaging the gradients of the corresponding inner maximization function. The gradients of the inner maximization function are approximated via the aforementioned gradient approximation method. Finally, the overall procedures for solving Eq. (5) are presented as Algorithm 1. The effect of the robust error-minimizing noise generated based on Algorithm 1 is illustrated in Fig. 2, which verifies the ability of error-minimizing noise of suppressing training loss in adversarial training.

heading: 5 EXPERIMENTS
text: In this section, we conduct comprehensive experiments to verify the effectiveness of the robust error-minimizing noise in data privacy protection.

heading: 5.1 EXPERIMENT SETUP
text: Datasets. Three benchmark computer vision datasets, CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), and an ImageNet subset (consists of the first 100 classes) (Russakovsky et al., 2015), are used in our experiments. The data augmentation technique (Shorten & Khoshgoftaar, 2019) is adopted in every experiment. For the detailed settings of the data augmentation, please see Appendix A.2.
Robust error-minimizing noise generation. Following Huang et al. (2021), we employ ResNet-18 (He et al., 2016) as the source model f ′θ for training robust error-minimizing noise generator with Eq. (5). The L∞-bounded noises ‖δu‖∞ ≤ ρu and ‖δa‖∞ ≤ ρa are adopted in our experiments. The detailed training settings are presented in Appendix A.3.1.
Baseline methods. The proposed robust error-minimizing noise is compared with two state-of-theart data protection noises, the error-minimizing noise (Huang et al., 2021) and the targeted adversarial poisoning noise (Fowl et al., 2021b). See Appendix A.3.2 for the noise generation procedures.
Model training. We follow Eq. (2) to conduct adversarial training (Madry et al., 2018) on data protected by different defensive noises with different models, including VGG-16 (Simonyan & Zisserman, 2014), ResNet-18, ResNet-50 (He et al., 2016), DenseNet-121 (Huang et al., 2017), and wide ResNet-34-10 (Zagoruyko & Komodakis, 2016). Similar to that in training noise generator, we also focus on L∞-bounded noise ‖ρa‖∞ ≤ ρa in adversarial training. Note that when ρa takes 0, the adversarial training in Eq. (2) degenerates to the ERM training in Eq. (1). The detailed training settings are given in Appendix A.4.
Metric. We use the test accuracy to assess the privacy protection ability of the noise. A low test accuracy suggests that the model learned little knowledge from the training data, which thus implies a strong privacy protection ability of the noise.

heading: 5.2 EFFECTIVENESS OF ROBUST ERROR-MINIMIZING NOISE
text: In this section, we study the effectiveness of robust error-minimizing noise from different aspects. We first visualize different defensive noises and the correspondingly crafted examples in Fig. 3. More visualization results can be found in Appendix B.
Different adversarial training perturbation radius. We first add different defensive noises to the entire training set. The defensive perturbation radius ρu of every noise is set as 8/255. Adversarial training is then conducted on both clean data and modified data with ResNet-18 models and different adversarial training perturbation radii ρa. Table 1 reports the accuracies of the trained models on
clean test data. We have also conducted experiments on noises that generated with a larger defensive perturbation radius ρu = 16/255, and the results are presented in Table 7 in Appendix C.
As shown in Table 1, when the adversarial training perturbation radius increases, the test accuracy of the model trained on unlearnable data and targeted adversarial poisoned data rapidly increases. On the other hand, the robust error-minimizing noise can always significantly reduces the test accuracy even when adversarial training presents. Furthermore, in the extreme case when the model trained on unlearnable data and targeted adversarial poisoned data achieve the same performance as that trained on clean data, the robust error-minimizing noise can still reduce the test accuracy of the model trained on robust unlearnable data by around 20% to 40%. These results demonstrate the effectiveness of the robust error-minimizing noise against adversarial learners.
Different protection percentages. We then study a more challenging as well as more realistic learning scenario, where only a part of the data are protected by the defensive noise, while the others are clean. Specifically, we randomly select a part of the training data from the whole training set, adding defensive noise to the selected data, and conduct adversarial training with ResNet-18 on the mixed data and the remaining clean data.The defensive perturbation radius for every noise is set as 8/255, while the adversarial perturbation radius ρa of the robust error-minimizing noise is set as 4/255. The accuracies on clean test data are reported in Table 2. The difference between the test accuracies on mixed data and clean data reflects the knowledge gained from the protected training data. We have also conducted experiments with noises that generated with a larger defensive perturbation radius ρu = 16/255. The results are reported in Table 8 in Appendix C.
Table 2 shows that when the adversarial training perturbation radius is small, robust errorminimizing noise can effectively protect the selected part of the data. However, when a large adversarial training perturbation radius presents, the protection becomes worthless. This suggests that to protect data privacy against adversarial training with a perturbation radius ρa, one has to set the defensive perturbation radius ρu of the robust error-minimizing noise to a value that is relatively larger than ρa. Table 2 shows that as the data protection percentage decreases, the performance of the trained model increases. This suggests that the model can learn more knowledge from more clean data, which coincides with intuition. Nevertheless, Table 2 further shows that the privacy preservation ability of the robust error-minimizing noise is stronger than that of the error-minimizing noise and the targeted adversarial poisoning noise. This demonstrates that the robust error-minimizing noise is still more favorable than other types of defensive noise when adversarial training presents.
Different model architectures. So far, we have only conduct adversarial training with ResNet18, which is as same as the source model in the defensive noise generation. We now evaluate the effectiveness of the robust error-minimizing noise under different adversarial learning models. Specifically, we conduct adversarial training with a perturbation radius 4/255 and five different types of model, including VGG-16, ResNet-18, ResNet-50, DenseNet-121, and wide ResNet-34-10, on data that is protected by noise generated via ResNet-18. Table 3 presents the test accuracies of the
trained models on CIFAR-10 and CIFAR-100. We have also conducted experiments on noises that generated with a larger defensive perturbation radius ρu = 16/255. See Table 9 in Appendix C.
Table 3 shows the robust error-minimizing noise generated from ResNet-18 can effectively protect data against various adversarially trained models. However, when the DenseNet-121 model presents, the robust error-minimizing noise could not achieve the same protection performance as that in other scenarios. This may partially be attributed to the limitation of DenseNet-121 itself in adversarial training, as it could not achieve the same generalization ability as those other models on clean data. We will leave further studies on this phenomenon in future works. Nevertheless, the experiment results still demonstrate the effectiveness of robust error-minimizing noise in most cases.
Ablation study on the EOT technique. In Section 4.3, we propose to employ the EOT technique to enhance the stability of the robust errorminimizing noise in adversarial training. Here, we empirically investigate the effect of EOT. Specifically, we conduct adversarial training with ResNet-18 on datasets protected by robust error noises that are generated with and without the EOT technique, respectively. The accuracies of the trained models on clean test data are reported in Table 4. The results show that without EOT, the generated noise gains no protection ability, which in
turn justifies the necessity of employing EOT during the robust error-minimizing noise generation.

heading: 6 CONCLUSION AND FUTURE WORKS
text: This paper proposes robust error minimizing noise, the first defensive noise that can protect data from being learned by unauthorized adversarial training. We deduce that the error-minimizing noise proposed by Huang et al. (2021) could not prevent unauthorized model training mainly because it can not suppress the training loss in adversarial training based on empirical study. Motivated by this, the robust error-minimizing noise is introduced to effectively reduce the adversarial training loss and thus suppress the learnable knowledge of data in adversarial training. Inspired by the adversarial training loss and the vanilla error-minimizing noise generation, a min-min-max problem is designed for training the robust error-minimizing noise generator, where the expectation over transform technique is adopted to enhance the stability of the generated noise. An important future direction is to establish theoretical foundations for the effectiveness of robust-error minimizing noise. Another interesting direction is to design more efficient robust error-minimizing noise generation methods.

heading: A EXPERIMENT DETAILS
text: This section provides the experiment details omitted from Section 5.

heading: A.1 HARDWARE DETAILS
text: The experiments on CIFAR-10 and CIFAR-100 are conducted on 1 GPU (NVIDIA® Tesla® V100 16GB) and 10 CPU cores (Intel® Xeon® Processor E5-2650 v4 @ 2.20GHz).
The experiments on ImageNet are conducted on 4 GPU (NVIDIA® Tesla® V100 16GB) and 40 CPU cores (Intel® Xeon® Processor E5-2650 v4 @ 2.20GHz).

heading: A.2 DATA AUGMENTATION
text: We use different data augmentations for different datasets. For CIFAR-10 and CIFAR-100, we perform data augmentation via random flipping, padding 4 pixels on each side, random cropping to 32 × 32 size, and rescaling per pixel to [−0.5, 0.5] for each image. For the ImageNet subset, we perform data augmentation via random cropping, resizing to 224 × 224 size, random flipping, and rescaling per pixel to [−0.5, 0.5] for each image.

heading: A.3 DEFENSIVE NOISE GENERATION
text: Our experiments involve three types of defensive noise, the proposed robust error-minimizing noise, and two baseline methods, error-minimizing noise and adversarial poisoning noise.

heading: A.3.1 ROBUST ERROR-MINIMIZING NOISE
text: Following Huang et al. (2021), we employ ResNet-18 (He et al., 2016) as the source model f ′ for the robust error-minimizing noise generation. The L∞-bounded noises ‖δu‖∞ ≤ ρu and ‖δa‖∞ ≤ ρa are adopted in our experiments, in which the defensive perturbation radius ρu and adversarial perturbation radius ρa can take various values. The settings of PGD for solving the inner minimization and maximization problems in Eq. (5) are presented in Table 5.
For CIFAR-10 and CIFAR-100, each source model is trained with SGD for 5, 000 iterations, with a batch size of 128, a momentum factor of 0.9, a weight decay factor of 0.0005, an initial learning rate of 0.1, and a learning rate scheduler that decay the learning rate by a factor of 0.1 every 2, 000 iterations. For EOT, the data transformation T is set as the data augmentation of the corresponding dataset, and the repeatedly sampling number for expectation estimation is set as 5.
Besides, for the ImageNet subset, each source model is trained with SGD via Eq. (5) for 3, 000 iterations, with a batch size of 128, a momentum factor of 0.9, a weight decay factor of 0.0005, an initial learning rate of 0.1, and a learning rate scheduler that decay the learning rate by a factor of 0.1 every 1, 200 iterations. For EOT, the data transformation T is set as the data augmentation of the corresponding dataset, and the repeatedly sampling number for expectation estimation is set as 4.

heading: A.3.2 BASELINE METHODS IMPLEMENTATIONS
text: Two baseline methods are adopted in our experiments as comparisons, including the errorminimizing noise method and the targeted adversarial poisoning noise method. Every method is reproduced on our own.
Error-minimizing noise (Huang et al., 2021). We follow Eq. (3) to train the error-minimizing noise generator. The error-minimizing noise is then generated with the trained noise generator. ResNet-18 model is used as the source model f ′. PGD is employed for solving the inner minimization problem in Eq. (3), where the settings of PGD is presented in Table 5. Other hyperparameters for training the noise generator are set the same as that for the robust error-minimizing noise generator in the previous section.
Targeted adversarial poisoning noise (Fowl et al., 2021b). This type of noise is generated via conducting targeted adversarial attack to the model that is trained on clean data, in which the generated adversarial perturbation is used as the adversarial poisoning noise. Specifically, given a fixed model f0 and a sample (x, y), the targeted adversarial attack will generate noise via solving the problem arg max‖δu‖≤ρu `(f0(x+ δu), g(y)), where g is a permutation function on the label space Y . PGD with differentiable data augmentation (Geiping et al., 2021) is employed for solving the above problem. The hyper-parameters for the PGD is given in Table 5.

heading: A.3.3 TIME COSTS OF NOISE GENERATIONS
text: We calculate the time costs for training different types of noise generators on different datasets. The results are reported in the following Table 6.

heading: A.4 MODEL TRAINING DETAILS
text: We follow Eq. (2) to perform adversarial training (Madry et al., 2018). Similar to that in training noise generator, we also focus on L∞-bounded noise ‖ρa‖∞ ≤ ρa in adversarial training. In every experiment, the model is trained with SGD for 40, 000 iterations, with a batch size of 128, a momentum factor of 0.9, a weight decay factor of 0.0005, an initial learning rate of 0.1, and a learning rate scheduler that decays the learning rate by a factor of 0.1 every 16, 000 iterations. For CIFAR-10 and CIFAR-100, the steps number Ka and the step size αa in PGD are set as 10 and ρa/5. For the ImageNet subset, the steps number Ka and the step size αa are set as 8 and ρa/4.

heading: B NOISE VISUALIZATION
text: This section presents more visualization results of different defensive noises. The defensive perturbation radius ρu of every noise is set as 8/255, while the adversarial perturbation radius of the robust error-minimizing noise is set as 4/255.

heading: CIFAR-10
text: CIFAR-100

heading: ImageNet subset

heading: C MORE EXPERIMENTS RESULTS
text: This section collects the additional experiment results for Section 5.2.
Different adversarial training perturbation radius.
Different data protection percentages.

heading: Different model architectures.
text: The evolution of the train and test accuracies along the adversarial training process. The curves of train and test accuracies on data protected by different defensive noises are drawn in Fig. 7. The figure shows that robust error-minimizing noise can effectively protect data from being learned along the adversarial training process, while other existing approaches could not. This result further justifies the effectiveness of our proposed method.
The selections of adversarial perturbation radius ρa in robust error-minimizing noise generation. We conduct adversarial training against the robust error-minimizing noises generated with a fixed defensive perturbation radius ρu and different adversarial perturbation radii ρa. The results are presented Fig. 8, which suggests that setting the adversarial perturbation radius to be half of the defensive perturbation radius would help the noise achieve consistent protection ability against adversarial training with different perturbation radii.

heading: D ADDITIONAL EXPERIMENTS OF FACIAL RECOGNITION
text: We have conducted a preliminary study on applying the robust error-minimizing noise to facial recognition. The goal is to protect a part of identities being recognized in facial recognition.
Specifically, we conduct adversarial training on the WebFace dataset (Yi et al., 2014) with ResNet18. The WebFace dataset consists of face images of 10,575 identities, in which 20% of the examples are separated as test data. We randomly select 50 identities for protection. To generate protective noise for the selected identities, we leverage 100 additional identities from the CelebA dataset (Liu et al., 2015) and train a ResNet-18 noise generator on the combined facial dataset of 150 identities. For each facial image from the WebFace dataset or the CelebA subset, we perform data augmentation via resizing to 112× 112 size, random flipping, and rescaling per pixel to the range [−0.5, 0.5]. We follow the similar procedures described in Appendix A.3 to generate different types of defensive noise. The ResNet-18 model is used as the source model f ′. Each noise generator is trained for 4, 000, with a batch size of 32, a momentum factor of 0.9, a weight decay factor of 0.0005, an initial learning rate of 0.1, and a learning rate scheduler that decay the learning rate by a factor of 0.1 every 1, 600 iterations. The repeatedly sampling number for the expectation estimation in EOT is set as 4. The hyper-parameters for PGD are presented in Table 10. The experiment results are presented in Fig. 9.
Table 10: The settings of PGD for the noise generations in facial recognition experiments.
Noise Type ρu αu Ku ρa αa Ka
EM 16/255 ρu/4 8 - - - AP 16/255 ρu/50 100 - - -
REM 16/255 ρu/4 7 8/255 ρa/4 7
Protected Unprotected 0
20
40
60
80
C la
ss ifi
ca tio
n Ac
cu ra
cy (%
)
ERM Training Clean EM TAP REM
Protected Unprotected 0
10
20
30
40
50
60
C la
ss ifi
ca tio
n Ac
cu ra
cy (%
)
Adversarial Training, u=2/255 Clean EM TAP REM
Figure 9: Facial recognition accuracy (%) on the protected and unprotected data. For every type of noise, the defensive perturbation radius is set as 16/255. The adversarial perturbation radius of the robust error minimizing noise is set as 8/255. ERM training and adversarial training with a perturbation radius ρa = 2/255 are conducted on the dataset.
From Fig. 9, one can be found that when slight adversarial training presents, robust error-minimizing noise can still effectively protecting facial data from being learned. This demonstrates the feasibility of applying robust error-minimizing noise in real-world scenarios. However, we also acknowledge that currently, the robust error-minimizing noise is unstable in facial recognition. For example, generating robust error-minimizing noises follows the same procedures with different random seeds may lead to completely different data protection levels. This may be due to the insufficiency of data. Usually, there are only 10∼ 20 face images for an identity in the WebFace or CelebA, which may be insufficient for generating robust error-minimizing noise with strong protection ability. We propose to conduct comprehensive studies for robust error-minimizing noise on face data in the future.

Review summaries:
This paper studies how to make data unlearnable towards adversarial training. In prior works, including dataset poisoning, no methods can prevent data from being learned in adversarial training. This paper proposed a robust error-minimizing noise to solve this task. 

Based on a previous work (Huang et al. 2021), this paper proposed a method, called REM, that generates makes a dataset "unlearnable" to adversarial training. Experiments are conducted to validate the effectiveness of REM under various settings, including (1) different adversarial training perturbation radiuses, (2) different proportions of unlearnable data, (3) different model architectures, on different common datasets, including CIFAR-10, CIFAR-100, and ImageNet.

The paper proposes a min-min-max formulation to generate robust unlearnable examples in order to protect data privacy in adversarial learning. The basic idea of this paper builds upon a previous paper "Unlearnable examples: Making personal data unexploitable" where the error-minimization noise is generated to reduce the training loss such that the model performs badly on the clean data. This paper finds that such unlearnability is fragile in adversarial training and fragile to minor data augmentations. Therefore, it proposes to generate more robust unlearnable samples by considering adversarial loss and data augmentation in the noise generation process, and it shows the generated noise can effectively reduce the quality of the trained model through extensive empirical study.  

Error minimization noise was first proposed so that when a dataset is released to the public, conventional empirical risk minimization cannot learn a good model from the perturbed dataset. However, error minimizing noise is only effective to the extent that adversarial training is not used. Adversarial training was identified as an effective method of overcoming the error minimization noise in prior work. In this paper, the authors proposed a variant of the error minimization noise, which continues to be robust even in the presence of adversarial training. Specifically, they generat the adversarial noise with a similar method as the original error minimization paper, but calculate the gradients with respect to the attacked input as opposed to the clean input.