Title: What Can We Learn from Unlearnable Datasets?
Venue: NeurIPS 2023
Accepted: yes
Keywords: data poisoning

poisons

unlearnable dataset

data protection

imperceptible perturbations

adversarial machine learning
Abstract: In an era of widespread web scraping, unlearnable dataset methods have the potential to protect data privacy by preventing deep neural networks from generalizing. But in addition to a number of practical limitations that make their use unlikely, we make a number of findings that call into question their ability to safeguard data. First, it is widely believed that neural networks trained on unlearnable datasets only learn shortcuts, simpler rules that are not useful for generalization. In contrast, we find that networks actually can learn useful features that can be reweighed for high test performance, suggesting that image protection is not assured. Unlearnable datasets are also believed to induce learning shortcuts through linear separability of added perturbations. We provide a counterexample, demonstrating that linear separability of perturbations is not a necessary condition. To emphasize why linearly separable perturbations should not be relied upon, we propose an orthogonal projection attack which allows learning from unlearnable datasets published in ICML 2021 and ICLR 2023. Our proposed attack is significantly less complex than recently proposed techniques.

What Can We Learn from Unlearnable Datasets?

Abstract

In an era of widespread web scraping, unlearnable dataset methods have the potential to protect data privacy by preventing deep neural networks from generalizing. But in addition to a number of practical limitations that make their use unlikely, we make a number of findings that call into question their ability to safeguard data. First, it is widely believed that neural networks trained on unlearnable datasets only learn shortcuts, simpler rules that are not useful for generalization. In contrast, we find that networks actually can learn useful features that can be reweighed for high test performance, suggesting that image protection is not assured. Unlearnable datasets are also believed to induce learning shortcuts through linear separability of added perturbations. We provide a counterexample, demonstrating that linear separability of perturbations is not a necessary condition. To emphasize why linearly separable perturbations should not be relied upon, we propose an orthogonal projection attack which allows learning from unlearnable datasets published in ICML 2021 and ICLR 2023. Our proposed attack is significantly less complex than recently proposed techniques. 1

1 Introduction

Deep learning is fueled by an abundance of data, the collection of which is largely unregulated [17, 20, 14, 32]. Images of human faces [9], artwork [1], and text [22, 24] are increasingly scraped at scale without consent. In an attempt to prevent the unauthorized use of data, unlearnable dataset methods make small perturbations to data so that deep neural networks (DNNs) trained on the modified data result in poor test accuracy [10, 7, 6, 36, 27]. The idea is that if generalization performance is harmed by incorporating the modified, 'unlearnable' data, third parties will be disincentivized from scraping it. Unlearnable dataset methods answer the following question: How should one imperceptibly modify the clean training set to cause the largest generalization gap?

In this paper, we analyze properties of unlearnable dataset methods in order to assess their future viability and security promises. Currently, unlearnable dataset methods are unlikely to be used for safeguarding public data because they require perturbing a large portion ( e . g . more than 50% ) of the training set [10, 27] and adversarial training is a principled attack [31]. Additionally, published data is immutable and must withstand current and future attacks [23]. While there are a number of reasons why these unlearnable dataset methods are unlikely to be employed, our results shed light on privacy implications and challenge common hypotheses about how they work. We make several findings by analyzing a number of unlearnable datasets developed from diverse objectives and theory. In particular,

We demonstrate that, in many cases, neural networks can learn generalizable features from unlearnable datasets. Our results imply that while resulting test accuracy may be low, DNNs may still learn useful features from protected data.

Canbeusedtolearnfeatureswhichgeneralizeto..

CleanDataset

Perturbation

UnlearnableDataset

Mayberecoveredfrom...

NeednotbeLinearlySeparable!

We challenge the common belief that unlearnable datasets work due to linearly separable perturbations. We construct a counterexample, suggesting that there is room for new unlearnable dataset methods that evade common attacks.

Inspired by our results on linear separability of perturbations, we present a new Orthogonal Projection attack which allows learning from unlearnable datasets perturbed by classwise, linearly separable perturbations. Our attack demonstrates that class-wise applied perturbations do not protect data from being learned. Our attack is often more effective than adversarial training, at a fraction of the computational cost.

Terminology Data poisoning methods that perturb the entire training dataset, and which we refer to as 'unlearnable datasets' or simply 'poisons', are also known as availability attacks [7, 36], generalization attacks [37], delusive attacks [31], or simply unlearnable examples [35]. Throughout this work, unlearnable datasets are considered defenses, given that their primary use case is to prevent the exploitation of data. Aiming to learn from unlearnable datasets is considered an attack.

2 Related Work

Unlearnable Datasets One of the earliest instances of a data poisoning attack intended to reduce overall test performance is from [2], who optimize a single sample that corrupts SVM training. Poisoning a convex model like SVM or logistic regression can be solved exactly, but poisoning deep neural networks is more challenging. A number of approaches have been proposed, validated primarily by empirical evidence due to the complexity of the objective (See Eq. 1 and 2). [5] propose using an auto-encoder-like network to generate perturbations against a victim classifier, [37] use NTKs to understand network predictions and optimize perturbations to produce misclassifications, [36] generate linearly separable perturbations, [10, 28] optimize error-minimizing noise whereas [7] optimize error-maximizing noise, and [27] generate autoregressive patterns, among other methods. The diversity of approaches motivates us to understand whether these unlearnable datasets share any properties in common, as we explore in Section 4.3.

Privacy Vulnerabilities In the context of image classification, unlearnable datasets are said to protect privacy by ensuring the modified data is not used or included as part of a larger dataset. That is, training on only poisoned data should prevent test set generalization and training on a dataset consisting of clean and poisoned data should be no better than training just on the clean portion. If poisoned training data is rendered useless for test set generalization, the data will not be used. In this way, only the initial owner can train on the original, unperturbed data and achieve high generalization performance. In recent work, [19] introduce the ability to 'lock' and 'unlock' training data by leveraging a class-wise perturbation, removable only by someone with knowledge of the exact perturbation. Unfortunately, as we empirically demonstrate in Section 4.4, class-wise perturbations can give a false sense of security. An argument against the use of poisoning for data privacy is the unavoidable reality that published data is immutable, and thus must withstand current and future methods which may exploit the data [23]. While it remains possible to recover relatively high test

accuracy from unlearnable datasets, our experiments also suggest that there is not yet a way to restore test accuracy to levels seen during clean training. Unlearnable datasets are known to be vulnerable to adversarial training [10, 31]. By formalizing unlearnable datasets as finding the worst-case training data within a ∞ -Wasserstein ball, [31] find that adversarial training [16] is a principled defense and can recover a high degree of accuracy by preventing models from relying on non-robust features. [28] challenge this assertion and optimize perturbations designed to degrade model performance under adversarial training. In Section 4.4, we show that adversarial training remains a strong defense against unlearnable datasets. In fact, we find that hyperparameter changes to the adversarial training recipe allows learning from Robust Unlearnable Examples [28], demonstrating the brittleness of unlearnable datasets.

How Unlearnable Datasets Work There are many explanations for how unlearnable datasets prevent networks from test set generalization: error-minimizing perturbations cause overfitting [10], error-maximizing noise stimulates learning non-robust features [7], convolutional layers are receptive to autoregressive patterns [27], and more. There are a variety of explanations because different methods arose from different optimization objectives and theory. But the leading explanation comes from Yu et al. [36], who find near perfect linear separability of perturbations for all the unlearnable datasets they consider. They explain that unlearnable datasets cause learning shortcuts due to linear separability of perturbations. In Section 4.3, we find a counterexample, demonstrating that while linear separability of perturbations may certainly be a property that helps unlearnable datasets function, the property is not necessary.

3 Problem Setting

We consider the problem of creating a clean-label unlearnable dataset in the context of a K -way image classification task, following [10]. We denote the clean training and test datasets as D train and D test , respectively.

Suppose there are n samples in a clean training set, i.e. D train = { ( x i , y i ) } n i =1 where x i ∈ R d are the inputs and y i ∈ { 1 , ..., K } are the labels. We consider a unlearnable dataset, denoted ˜ D train = { ( x ′ i , y i ) } n i =1 where x ′ i = x i + δ i is the perturbed or poisoned version of the example x i ∈ D train and where δ i ∈ ∆ ⊂ R d is the perturbation. The set of allowable perturbations, ∆ , is typically an ℓ p -ball with radius ϵ where ϵ is small enough that δ is imperceptible.

Unlearnable datasets are created by applying a perturbation to a clean image in either a class-wise or sample-wise manner. When a perturbation is applied class-wise, every sample of a given class is perturbed in the same way. That is, x ′ i = x i + δ y i and δ y i ∈ ∆ C = { δ 1 , ..., δ K } . When samples are perturbed in a sample-wise manner, every sample has a unique perturbation.

All unlearnable dataset methods aim to solve the following bi-level maximization:

Eq. 2 describes the process of training a model on unlearnable data, where θ denotes the model parameters. Eq. 1 states that the unlearnable data should be chosen so that the trained network has high test loss, and thus fails to generalize to the test set.

4 Experiments

4.1 Datasets and Training Settings

For all of our experiments, we make use of open-source unlearnable datasets: From Unlearnable Examples [10], we use their sample-wise and class-wise poisons. From Adversarial Poisoning [7], we use their targeted PGD attack poison. From Autoregressive Poisoning (AR) [27], Neural Tangent Generalization Attacks (NTGA) [37], Robust Unlearnable Examples [28], and Linearly Separable

Perturbations (LSP) [36], we use their main poison. For One-pixel Shortcut [35], we use their OPS and CIFAR-10-S poisons, but we refer to their CIFAR-10-S poison as OPS+EM for ease of attribution. We also include two class-wise Regions-4 and Random Noise poisons which contain randomly sampled perturbations, following [26]. To poison a K -class dataset in a class-wise manner, we need K perturbation vectors. For Regions-4 , each perturbation is independently created by sampling 4 vectors of size 3 from a Bernoulli distribution. Each vector is then scaled to lie in the range [ -8 255 , 8 255 ] . Finally, each of the 4 vectors are repeated along height and width dimensions to create patches of size 16 × 16 , which are then arranged side by side to achieve a shape of 32 × 32 . For Random Noise perturbations, we sample an i.i.d. vector, one entry per pixel, from a Bernoulli and scale perturbations to fit the imperceptibility constraint. See Appendix A.5 for image samples for all unlearnable datasets we consider.

Results in this section are for CIFAR-10 [13], and additional results for SVHN [18], CIFAR-100 [13], and ImageNet [25] Subset are in Appendix A.2.2 and A.4.2. Unlearnable datasets constructed with class-wise perturbations are prefixed with ◦ , otherwise the perturbations were added sample-wise. For the imperceptibility constraint, the AR ( ℓ 2 ) dataset contains perturbations δ of size ∥ δ ∥ 2 ≤ 1 . LSP has perturbations of size ∥ δ ∥ 2 ≤ ϵ ′ √ d , where d is the image dimension and ϵ ′ = 6 255 . OPS and OPS+EM contain unbounded perturbations. All other datasets contain perturbations of size ∥ δ ∥ ∞ ≤ 8 255 .

We primarily use the ResNet-18 (RN-18) [8] architecture for training on the above datasets. We include results for additional network architectures including VGG-16 [29], GoogleNet [30], and ViT [4] in Appendix A.2.1 and A.4.1. Training hyperparameters can be found in Appendix A.1.

4.2 DNNs Can Learn Useful Features From Unlearnable Datasets

Unlearnable datasets are meant to be unlearnable; models trained on unlearnable datasets should not be capable of generalizing to clean test data. Presumably, the modified, unlearnable dataset protects the clean version of the dataset from unauthorized use in this fashion. But by reweighting deep features, we find that DNNs are still able to recover some generalizable features from unlearnable datasets.

DFR Method The technique of training a new linear classification layer on top of a fixed feature extractor is known as Deep Feature Reweighting (DFR) [12] and it is a method for learning from data with spurious correlations, where predictive features in the train distribution are not present in the test

0

10

20

30

40

50

60

0

10

20

30

40

50

60

70

80

90

100

(a)

Epoch

CIFAR-10 Test Accuracy

Adversarial Poisoning

DFR Adversarial Poisoning

0

10

20

30

40

50

60

0

10

20

30

40

50

60

70

80

90

100

(b)

Epoch

CIFAR-10 Test Accuracy

0

10

20

30

40

50

60

0

10

20

30

40

50

60

70

80

90

100

(c)

Epoch

CIFAR-10 Test Accuracy

AR

0

10

20

30

40

50

60

0

10

20

30

40

50

60

70

80

90

100

(d)

Epoch

CIFAR-10 Test Accuracy

NTGA

DFR NTGA

distribution. We borrow the DFR method to better understand the utility of features learned during poison training. The higher the test accuracy after DFR, the more likely it is that the model has learned features present in the original clean data. If DFR works well on poison-trained networks, this would suggest that even poisoned weights contain a semblance of relevant information for generalization.

To evaluate the extent that generalizable features are learned, we start by saving network weights at every epoch of poison training. In the context of image classification, network weights consist of a feature extractor followed by a fully-connected classification layer mapping feature vectors to class logits. Next, for each checkpoint, we utilize a random subset of 5 , 000 clean CIFAR-10 training samples ( 10% of the original training set) to train a new classification head (and keep the feature extractor weights fixed). Finally, we plot test accuracy in Figure 2 and evaluate the maximum CIFAR-10 test accuracy achieved through DFR for each dataset and report results in Table 1. As a baseline, we train a classification head on a randomly initialized RN-18 feature extractor and find that these random features can be reweighted to achieve 35 . 97% test accuracy.

DFR Results We find that, to different extents, DNNs do learn generalizable features from unlearnable datasets. Surprisingly, on Adversarial Poisoning and NTGA , RN-18 features actually improve throughout training , as we show in Figure 2 (a) and ( d ), despite low test accuracy during poison training (solid lines in Figure 2 (a) and (d) ). On AR and ◦ Random Noise , test accuracy peaks early in training before dropping to random chance accuracy (solid lines in Figure 2 (b) and (c) ). Still, when early checkpoints are used, one can use features from the poison-trained feature extractors

to achieve high accuracy (dashed lines in Figure 2 (b) and (c) ). Accuracy curves for Adversarial Poisoning are unique because images are perturbed with error-maximizing noise, which correspond to actual features models use during classification [11]. In this case, DFR is reweighting useful, existing features for classification, leading to higher test accuracy. On the other hand, Random Noise and AR poisons do not perturb images with useful features; instead, both perturb with synthetic noise. In these cases, useful features are still learned during poison training, but only in the first epochs. As training progresses, the model checkpoints are continually corrupted by synthetic noise features which cannot be useful for classification despite reweighting.

As we show in Table 1, ◦ Unlearnable Examples are very effective at corrupting network weights during training; using DFR on checkpoints from poison training only yields a maximum test accuracy of 34 . 09% , nearly 2% worse than a randomly initialized feature extractor. Six of the eleven unlearnable datasets we consider yield checkpoints which achieve DFR test accuracies that are 10% or more higher than a randomly initialized feature extractor. Generally, it is interesting that unlearnable dataset methods with the lowest max DFR test accuracy are those which use error-minimizing perturbations in some capacity, e . g . Unlearnable Examples , Robust Unlearnable Examples , and OPS+EM . Analyzing test loss follows trends from test accuracy: Adversarial Poisoning , AR , NTGA , and Random Noise achieve lowest losses - and those poisoned checkpoints also have the highest DFR test accuracy in Table 1. More interestingly, we find that all poisoned models have a lower loss than the randomly initialized model. 2 This reinforces our claim that models learned useful features from poisoned data.

Overall, our results suggest that network weights during poison training are not fully corrupted in many cases. While poison-trained networks may be evaluated to have low test accuracy, we show that, for some unlearnable datasets, the networks have learned generalizable features which can be reweighted for high test performance. It is entirely possible that these features, which encode the original data amount to useful image information that the original data owner did not want incorporated into an ML system.

Privacy concerns are often cited as primary motivation for unlearnable datasets, yet our results raise concerns about this framing. A model trained on an apparently unlearnable dataset might have low test error, but this does not imply, as we show, that it does not contain usable information about the original data, and might not keep promises to protect data privacy.

4.3 Linearly Separable Perturbations Are Not Necessary

The work of Yu et al. [36] demonstrated that unlearnable datasets contain linearly separable perturbations and suggested that linear separability is a necessary property of unlearnable datasets. However,

by finding a counterexample, we confirm that linear separability is not necessary to have an effective unlearnable dataset.

Following [36], we train linear logistic regression models on image perturbations by subtracting the clean image from the perturbed image. Given the publication of new unlearnable datasets, we add to Yu et al.'s analysis by including AR , Robust Unlearnable Examples , and OPS poisons. It is possible to obtain image perturbations given our access to clean CIFAR-10, but should not be possible in practice if the clean version of the unlearnable dataset is unavailable to the public. We optimize the logistic regression weights ( 3072 × 10 parameters for CIFAR-10) using L-BFGS [15] for 500 steps using a learning rate of 0 . 5 . For reference, we also train a logistic regression model on clean CIFAR-10 image pixels and report the train accuracy in the first row of Table 2. For results on the linear separability of the perturbed images , see Appendix A.3.2.

While most current unlearnable datasets contain linearly separable perturbations, as was initially shown by Yu et al. [36], AR perturbations stand out as a counterexample in Table 2. After training on AR perturbations, a logistic regression model can only achieve 39 . 58% train accuracy while all other unlearnable datasets we tested contain perturbations that are almost perfectly linearly separable. Because most unlearnable dataset methods developed from diverse optimization objectives result in linearly separable perturbations, we posit that linear separability is an easy solution for optimization of Eq. 1 and 2 to find. For unlearnable datasets that are not optimized, class-wise perturbations are trivially linearly separable, so long as no two class perturbations are the same. Thus, although AR perturbations are not linearly separable, they remain separable by a simple 2-layer CNN [27], suggesting that simple perturbations may be a more accurate way of defining effective unlearnable perturbations.

Being the first counterexample of its kind, we believe there may be underexplored unlearnable dataset methods. This finding sheds light on the complexity of unlearnable datasets, whose behavior depends on choices of loss function, optimizer, and even network architecture. An important detail about AR perturbations is that they are not optimized - they are generated [27]. We leave to future work an investigation as to why most optimized perturbations in unlearnable datasets to date result in linear separability.

4.4 Orthogonal Projection for Learning From Datasets with Class-wise, Linear Perturbations

In this section, we develop a method to learn from unlearnable datasets. Given our understanding of linear separability from Section 4.3, learning the most predictive features in an unlearnable dataset can amount to learning the perturbation itself. We leverage this intuition in our Orthogonal Projection attack, which works by learning a linear model, then projecting the data to be orthogonal to linear weights. In other words, we remove the most predictive dimensions of the data. For unlearnable datsets perturbed by class-wise, linearly separable perturbations, the most predictive dimensions are easily found by a linear model. Attacks for learning from class-wise perturbed unlearnable datasets have included using diffusion models [3], adversarial training [10], and error-maximizing augmentations [21]. In contrast to these techniques, our method is simpler and computationally cheaper.

4.4.1 Orthogonal Projection Method

Unlearnable datasets with class-wise perturbations are so simple that visualizing the average image of a class can expose the class perturbation (see Figure 5). In order to adaptively learn these perturbations and remove them from poisoned data, we propose a method that first learns the simple perturbations and then orthogonally projects samples to omit them. Our Orthogonal Projection method is designed

CleanBatch

LSP Perturbation

LSP Batch

(e)

Class 1

Class 2

Class7

Class 1

Class 2

Class 7

OPS Perturbation

OPS Batch

Class 0

Class 1

Class 2

Class3

Class 4

Class5

Class 6

Class7

Class8

Class 9

LSP

(a)

OPS

Clean

to exploit class-wise perturbations by design and is meant to emphasize why class-wise perturbations should not be relied on for creating unlearnable datasets.

Given an unlearnable dataset, we train a simple logistic regression model on poison image pixels, in an attempt to optimize the most predictive linear image features. The result is a learned feature matrix W , on which we perform a QR decomposition such that W = QR , where Q consists of orthonormal columns. Image samples X from the unlearnable dataset can be orthogonally projected using the matrix transformation I -QQ T . Orthogonal projection ensures that the dot product of a row of X with every column of Q is zero, so that learned perturbations from the logistic regression model are not seen during the final training run on recovered data. The recovered image samples are then written as X r = ( I -QQ T ) X . Finally, we train the final network on recovered data, X r . A detailed algorithm is shown in Algorithm 1. We include additional intuition in Appendix A.4.3.

The added computational cost of Orthogonal Projection over standard training is only in training the logistic regression model and projecting the dataset, which can be done once. For CIFAR-10, we train the logistic regression model for 20 epochs using SGD with an initial learning rate of 0 . 1 , which decays by a factor of 10 on epochs 5 and 8 . We find that training the linear model for longer is often detrimental, potentially because of overfitting.

4.4.2 Orthogonal Projection Results

We compare our attack against ℓ ∞ adversarial training, as previous work has shown that adversarial training is a principled defense [31] against unlearnable datasets. For adversarial training, we use a 3 -step PGD adversary with a perturbation radius of 8 255 in ℓ ∞ -norm. Note that large adversarial training perturbation radii harm clean accuracy [33]. We use SGD with momentum of 0 . 9 and an initial learning rate of 0 . 1 , which decays by a factor of 10 on epoch 75 , 90 , and 100 .

We find our method is more effective than adversarial training when learning from unlearnable datasets corrupted in a class-wise manner. In Table 3, our Orthogonal Projection attack leads to greater test accuracy on all class-wise poisons prefixed by ◦ . By visualizing the learned weights from

the linear model in Figure 3, we see that the class-wise, linearly separable perturbations are recovered. Projecting orthogonal to these vectors effectively removes those features from the unlearnable dataset, allowing training to proceed without predictive, but semantically meaningless features. It should not be surprising that Orthogonal Projection is also effective against OPS (ICLR 2023), given that all images of a class have the same pixel modified. In Figure 3, we visualize the linear model weights and show that it is able to pinpoint the modified class pixel, whose influence is effectively removed after projection in our method. We visualize additional linear model weights for other datasets in Appendix A.4.5. Orthogonal Projection is also surprisingly effective against NTGA (ICML 2021), such that an RN-18 model can achieve 82 . 21% test accuracy when trained on orthogonally projected NTGA images. For seven of the eleven unlearnable datasets we consider, Orthogonal Projection can achieve above 80% test accuracy without adversarial training. Unlike the linear model from Section 4.3 which is trained on only perturbations, the linear model that we train for Orthogonal Projection trains on perturbed images and the resulting data is no longer linearly separable. For this reason, Orthogonal Projection struggles against Adversarial Poisoning , AR , and Unlearnable Examples ; the training distribution of the linear model is more complex than just perturbations. Given that OPS+EM is a combination of OPS and class-wise error-minimizing noise from Unlearnable Examples , both Orthogonal Projection and adversarial training are ineffective. For both OPS and OPS+EM , adversarial training achieves approximately 12% accuracy, a dismal result that is expected given the unbounded OPS perturbations. A limitation of our approach is that, by orthogonally projecting data, we effectively remove K dimensions from the data manifold, where K is the number of classes in a dataset. While this may not be a problem for high-resolution images with tens of thousands of dimensions, this detail could impact applicability for low-resolution datasets.

The susceptibility of using class-wise perturbations to craft unlearnable datasets should be expected, given that every image of a class contains the same image feature that is perfectly predictive of the label. Class-wise unlearnable datasets contain simple features that are perfectly correlated with the label and the linear logistic regression model is able to optimize features resembling the original perturbations. By design, our orthogonal projection attack is most effective against class-wise perturbed unlearnable datasets. It serves as evidence that class-wise, linearly separable perturbations cannot be relied upon for protecting data.

5 Conclusion

We design experiments to test prevailing hypotheses about unlearnable datasets, and our results have practical implications for their use. First, while all unlearnable datasets cause low test accuracy for

trained models, we demonstrate that generalizable features can still be learned from data. In many cases, high test accuracy can be achieved from a feature extractor trained on unlearnable datasets. If data were modified for protection, an adversary may still be able to train a reasonable feature extractor, a harm for data thought to be secure. We advocate for our evaluation framework to be used for auditing whether new unlearnable datasets prevent useful features from being learned. Second, we find that the reason unlearnable datasets cause low test accuracy is not as simple as previously thought. AR perturbations serve as a counterexample to the hypothesis that unlearnable datasets contain linearly separable perturbations. Our finding that AR perturbations are not linearly separable suggests there could be underexplored methods for protecting data using imperceptible perturbations. Although unlearnable datasets need not have linearly separable perturbations, if they do have them applied in a class-wise manner, we present an Orthogonal Projection attack that can effectively remove them. While learning the perturbations from perturbed images is challenging, our results imply class-wise perturbations cannot not be relied upon for data protection. We can learn a significant amount from unlearnable datasets. DNNs can learn generalizable features from unlearnable datasets assumed to be protected, linear separability is not a necessary condition for preventing generalization from unlearnable datasets, and class-wise perturbations can be optimized against and effectively removed.

Acknowledgments and Disclosure of Funding

This material is based upon work supported by the National Science Foundation under Grant No. IIS-1910132 and Grant No. IIS-2212182, and by DARPA's Guaranteeing AI Robustness Against Deception (GARD) program under #HR00112020007. Tom Goldstein was supported by the ONR MURI program, the AFOSR MURI program, and the National Science Foundation (IIS-2212182 & 2229885). Vasu Singla was supported in part by the National Science Foundation under grant number IIS-2213335. Pedro Sandoval-Segura is supported by a National Defense Science and Engineering Graduate (NDSEG) Fellowship. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

References

Andy Baio. Exploring 12 million of the 2.3 billion images used to train stable diffusion's image generator, Sep 2022.

Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. In Proceedings of the 29th International Coference on International Conference on Machine Learning , ICML'12, page 1467-1474, Madison, WI, USA, 2012. Omnipress.

Hadi M. Dolatabadi, Sarah Erfani, and Christopher Leckie. The devil's advocate: Shattering the illusion of unexploitable data using diffusion models, 2023.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Ji Feng, Qi-Zhi Cai, and Zhi-Hua Zhou. Learning to confuse: generating training time adversarial data with auto-encoder. Advances in Neural Information Processing Systems , 32, 2019.

Liam Fowl, Ping-yeh Chiang, Micah Goldblum, Jonas Geiping, Arpit Bansal, Wojtek Czaja, and Tom Goldstein. Preventing unauthorized use of proprietary data: Poisoning for secure dataset release. arXiv preprint arXiv:2103.02683 , 2021.

Liam Fowl, Micah Goldblum, Ping-yeh Chiang, Jonas Geiping, Wojciech Czaja, and Tom Goldstein. Adversarial examples make strong poisons. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 30339-30351. Curran Associates, Inc., 2021.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Kashmir Hill. The secretive company that might end privacy as we know it, Jan 2020.

Hanxun Huang, Xingjun Ma, Sarah Monazam Erfani, James Bailey, and Yisen Wang. Unlearnable examples: Making personal data unexploitable. In International Conference on Learning Representations , 2021.

Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. Advances in neural information processing systems , 32, 2019.

Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness to spurious correlations, 2022.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4. International Journal of Computer Vision , 128(7):1956-1981, 2020.

Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical programming , 45(1-3):503-528, 1989.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations , 2018.

Clare McGlynn, Erika Rackley, and Ruth Houghton. Beyond 'revenge porn': The continuum of imagebased sexual abuse. Feminist legal studies , 25(1):25-46, 2017.

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.

Weiqi Peng and Jinghui Chen. Learnability lock: Authorized learnability control through adversarial invertible transformations. In International Conference on Learning Representations , 2022.

Vinay Uday Prabhu and Abeba Birhane. Large image datasets: A pyrrhic win for computer vision? 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) , pages 1536-1546, 2020.

Tianrui Qin, Xitong Gao, Juanjuan Zhao, Kejiang Ye, and Cheng-Zhong Xu. Learning the unlearnable: Adversarial augmentations suppress unlearnable example attacks. arXiv preprint arXiv:2303.15127 , 2023.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Evani Radiya-Dixit, Sanghyun Hong, Nicholas Carlini, and Florian Tramer. Data poisoning won't save you from facial recognition. In International Conference on Learning Representations , 2022.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. , 21(140):1-67, 2020.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision , 115:211-252, 2015.

Pedro Sandoval-Segura, Vasu Singla, Liam Fowl, Jonas Geiping, Micah Goldblum, David Jacobs, and Tom Goldstein. Poisons that are learned faster are more effective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , pages 198-205, June 2022.

Pedro Sandoval-Segura, Vasu Singla, Jonas Geiping, Micah Goldblum, Tom Goldstein, and David Jacobs. Autoregressive perturbations for data poisoning. Advances in Neural Information Processing Systems , 35:27374-27386, 2022.

Fu Shaopeng, He Fengxiang, Liu Yang, Shen Li, and Tao Dacheng. Robust unlearnable examples: Protecting data privacy against adversarial learning. In International Conference on Learning Representations , 2022.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1-9, 2015.

Lue Tao, Lei Feng, Jinfeng Yi, Sheng-Jun Huang, and Songcan Chen. Better safe than sorry: Preventing delusive adversaries with adversarial training. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 16209-16225. Curran Associates, Inc., 2021.

Andrew Tarantola. Getty images sues the maker of ai art generator stable diffusion over data scraping allegations, Jan 2023.

Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In International Conference on Learning Representations , 2019.

Rui Wen, Zhengyu Zhao, Zhuoran Liu, Michael Backes, Tianhao Wang, and Yang Zhang. Is adversarial training really a silver bullet for mitigating data poisoning? In The Eleventh International Conference on Learning Representations , 2023.

Shutong Wu, Sizhe Chen, Cihang Xie, and Xiaolin Huang. One-pixel shortcut: On the learning preference of deep neural networks. In The Eleventh International Conference on Learning Representations , 2023.

Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie-Yan Liu. Availability attacks create shortcuts. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining . ACM, aug 2022.

Chia-Hung Yuan and Shan-Hung Wu. Neural tangent generalization attacks. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 12230-12240. PMLR, 18-24 Jul 2021.

A Appendix

A.1 Training Hyperparameters

DNNs Can Learn Useful Features From Unlearnable Datasets In Section 4.2, we train a number of ResNet-18 (RN-18) [8] models on different unlearnable datasets with cross-entropy loss for 60 epochs using a batch size of 128 . We save checkpoints at every epoch of training. For our optimizer, we use SGD with momentum of 0 . 9 and weight decay of 5 × 10 -4 . We use an initial learning rate of 0 . 1 which decays using a cosine annealing schedule.

For training a new classification layer on feature extractor checkpoints, we use 5 , 000 random clean images from the original training set data. Note that 5 , 000 images is 10% of CIFAR-10 and CIFAR-100, but 3 . 9% of Imagenet subset, etc. Following [12], using the feature extractor, we extract embeddings from this clean subset of data and preprocess the embeddings to have mean zero and unit standard deviation. To retrain the last layer, we use the logistic regression implementation from scikit-learn ( sklearn.linear_model.LogisticRegression ).

Linearly Separable Perturbations Are Not Necessary In Section 4.3, for every unlearnable dataset, we first gather the set of perturbations by subtracting the clean image from the perturbed image. We zero-one normalize the perturbations before training a linear layer using L-BFGS [15] for 500 steps using a learning rate of 0 . 5 .

Orthogonal Projection for Learning From Datasets with Class-wise, Linear Perturbations In Section 4.4, for CIFAR-10, we train the logistic regression model for 20 epochs using SGD with an initial learning rate of 0 . 1 , which decays by a factor of 10 on epochs 5 and 8 (at epochs which are 0 . 5 and 0 . 75 through training). For CIFAR-100, we train the logistic regression model for 60 epochs due to the higher number of classes. After we orthogonally project the unlearnable data using the optimized weights, we train networks using the hyperparameters from checkpoint-training of Section 4.2.

A.2 Additional Section 4.2 Results: DNNs Can Learn Useful Features From Unlearnable Datasets

A.2.1 More Model Architectures for Section 4.2

We consider three more model architectures: VGG-16 [29], GoogLeNet [30], and ViT [4]. Our ViT uses a patch size of 4 . For RN-18 and VGG-16, feature vectors are 512 -dimensional. Feature vectors for GoogleNet are 1024 -dimensional. The ViT class token is 384 -dimensional.

In Table 4, we find that across architectures, Adversarial Poisoning data is easiest to extract generalizable features from. Surprisingly, ViT is most effective at learning generalizable features from all unlearnable datasets, achieving more than 7% test accuracy improvement over a randomly initialized ViT in all cases. For example, using only 5 , 000 clean CIFAR-10 samples can be used to achieve nearly 69% test accuracy, while using the same clean samples can only achieve 37 . 40% test accuracy on a randomly initialized ViT. The GoogleNet architecture weights are seemingly more easily corrupted during training; Max DFR Test Accuracy for Unlearnable Examples , AR , Robust Unlearnable , and other datasets is much lower than test accuracy from a finetuned randomly initialized GoogleNet. Interestingly, the randomly initialized GoogleNet feature extractor achieves the highest DFR test accuracy.

A.2.2 More Datasets for Section 4.2

We consider three additional base datasets for four unlearnable dataset methods. We use an ImageNet [25] subset of the first 100 classes, following [10]. The train split consists of 129 , 395 images, while the test split consists of 5 , 000 images.

Our SVHN [18], CIFAR100 [13], and Adversarial Poisoning ImageNet subset datasets contain perturbations of size ∥ δ ∥ ∞ ≤ 8 255 . Unlearnable Examples ImageNet [25] subset contains perturbations of size ∥ δ ∥ ∞ ≤ 16 255 , following their open-source repository. We generate the Adversarial Poisoning [7] ImageNet subset from published source code using 1 PGD restart, as opposed to 8 due to computation time. Our SVHN and CIFAR-100 Adversarial Poisoning datasets use 3 PGD restarts. On clean SVHN, CIFAR-100, and ImageNet subset, RN-18 achieves 96 . 33% , 74 . 14% , 78 . 92% test accuracy respectively.

In Table 5, we again show that Adversarial Poisoning unlearnable data can be easily used to extract generalizable features regardless of the underlying distribution (base dataset of SVHN, CIFAR-100, or ImageNet). For SVHN, Adversarial Poisoning is the only dataset from which the trained feature extractor performs better in Max DFR Test Accuracy ( 87 . 06% ) over a randomly initialized RN-18 ( 32 . 05% ). As mentioned in Section 4.2, error-minimizing perturbations of Unlearnable Examples tend to be most effective at corrupting weights during training, regardless of underlying finetune data.

A.2.3 Additional Plots for Section 4.2

We add results to the experiment from Figure 2. In Figure 4, unlearnable datasets sufficiently corrupt RN-18 weights during training and prevent DFR from recovering test accuracy.

0

10

20

30

40

50

60

0

10

20

30

40

50

60

70

80

90

100

(a)

Epoch

CIFAR-10 Test Accuracy

Unlearnable Examples

DFR Unlearnable Examples

0

10

20

30

40

50

60

0

10

20

30

40

50

60

70

80

90

100

(b)

Epoch

CIFAR-10 Test Accuracy

Robust Unlearnable

DFR Robust Unlearnable

0

10

20

30

40

50

60

0

10

20

30

40

50

60

70

80

90

100

(c)

Epoch

CIFAR-10 Test Accuracy

LSP

DFR LSP

0

10

20

30

40

50

60

0

10

20

30

40

50

60

70

80

90

100

(d)

Epoch

CIFAR-10 Test Accuracy

OPS+EM

0

10

20

30

40

50

60

0

10

20

30

40

50

60

70

80

90

100

(e)

Epoch

CIFAR-10 Test Accuracy

OPS

DFR OPS

0

10

20

30

40

50

60

0

10

20

30

40

50

60

70

80

90

100

(f)

Epoch

CIFAR-10 Test Accuracy

◦

Unlearnable Examples

DFR

◦

Unlearnable Examples

A.3 Additional Results for Section 4.3: Linearly Separable Perturbations Are Not Necessary

A.3.1 Other Background

A related unlearnable dataset introduces entangled features (EntF) [34] which is motivated by the separability of recent poisoning perturbations. However, separability is qualitatively evaluated through t-SNE visualizations, which is different from the separability experiment we perform. More specifically, t-SNE cluster separability should not be equated to the linear separability we measure in Table 2 because it is possible to have linearly separable data that, when plotted using t-SNE, appears not separable. In other words, the EntF poison from [34] could still contain linearly separable perturbations.

A.3.2 Evaluating Linear Separability of Poison Images

In Section 4.3, we document the linear separability of perturbations from various poisons, as in [36]. Poison images, on the other hand, behave slightly differently. In Table 6, we report logistic regression train accuracy on various CIFAR-10 poison images. We find that Unlearnable Examples, LSP, OPS+EM , and class-wise poisons have linearly separable poison images, but the remaining poisons we consider do not.

A.4 Additional Section 4.4 Results: Orthogonal Projection for Learning From Datasets with Class-wise, Linear Perturbations

A.4.1 ViT for Section 4.4

To evaluate recovered data from Orthogonal Projection, we consider an additional architecture: ViT. In Table 7, we train ViT with patch size of 4 on CIFAR-10 unlearnable datasets using different attacks. Our Orthogonal Projection method is competitive with adversarial training for all class-wise perturbed unlearnable datasets and most sample-wise perturbed datasets. Orthogonal Projection is the best performing method for OPS+EM and OPS (ICLR 2023). Note that OPS+EM and OPS are most difficult for adversarial training.

A.4.2 CIFAR-100 Dataset for Section 4.4

We consider an additional dataset, CIFAR-100, to evaluate Orthogonal Projection. We train a RN-18 on four unlearnable dataset methods. During the first step of Orthogonal Projection, we train the logistic regression model for 60 epochs using SGD with an initial learning rate of 0 . 1 , which decays by a factor of 10 on epochs 30 and 45.

In Table 8, we find that Orthogonal Projection performs better on class-wise unlearnable data than sample-wise perturbed data, as expected. At approximately the cost of standard training (Attack: None), Orthogonal Projection achives gains of more than 20% test accuracy for sample-wise perturbed data and more than 40% test accuracy for class-wise perturbed data.

A.4.3 Additional Intuition for Orthogonal Projection

Assume CIFAR-10 images of shape (3 , 32 , 32) . Each column i of W (optimized in Alg 1, Lines 1-4) is a 3072 -dimensional vector that represents the most predictive image feature for class i . This step serves as recovery of the perturbation. After the QR decomposition of W , Q consists of orthonormal columns that form a basis for the column space of W . When we say Orthogonal Projection 'ensures that the dot product of a row of X with every column of Q is zero,' (i.e., X r · Q = 0 ) this means that every recovered image vector does not contain any linearly separable component (i.e., does not contain any column of Q as a component). Alg. 1, Line 6 ensures image vectors and columns of Q are orthogonal and so the dot product is 0 . The 'recovered' data thus has 10 dimensions (approximations of the 10 perturbations) removed.

A.4.4 Subtracting a Class-wise Image

Given that the goal of Orthogonal Projection is to extract perturbations from poison images, it is reasonable to consider visualizing the average image of a class for class-wise poisons like LSP and OPS . In Figure 5, we see that class-wise average images somewhat reveal class-wise perturbations, but the results are not clear enough to be useful. In contrast, class-wise patters and clearly present in learned weights from logistic regression.

Class0Class1Class 2Class3Class4

LSP

OPS

Class0Class1Class2Class3Class4

Figure 5: Average class images display class-wise perturbations, but not as clearly as learned weights from logistic regression. We compare the average image of a class (Left) and learned weights of a logistic regression classifier (Right) trained on image pixels (the first step of Orthogonal Projection method) to for LSP and OPS Poisons. While average image of a class does reveal the class-wise perturbation (block pattern for LSP and one highlighted pixel for OPS ), the result is blurry and contains other semantic image features. Learned weights from Orthogonal Projection properly isolate the perturbation.

For training models on class-wise perturbed data, one might consider subtracting the class-wise average image from each class. However, simply subtracting this average class image from each image does not remove the poisoning effect. Additionally, because we do not know the true class at inference time, we cannot subtract the class image, resulting in a distribution mismatch between train and test sets. This trivial method of subtracting average class images is compared to Orthogonal Projection in Table 9.

A.4.5 Visualizing Additional Logistic Regression Weights

We visualize additional linear model weights (from the first step of Orthogonal Projection) for sample-wise perturbed unlearnable datasets in Figure 6, and for class-wise perturbed unlearnable datasets in Figure 7. We find that for Adversarial Poisoning , AR , and Robust Unlearnable the linear model learns features comparable to when trained on clean data. We posit that because the diversity of perturbations in these datasets is higher, the linear model struggles to find predictive features to project away. In contrast, for class-wise perturbed data, Figure 7, demonstrates that the linear model can recover features that resemble the original class-wise perturbation.

Class0

Class1

Class2

Class3

Class4

Class5

Class6

Class7

Class8

Class9

Unlearnable

Examples

Adversarial

Poisoning

AR(L2)

NTGA

Robust

Unlearnable

LSP

A.5 Samples from Unlearnable Datasets

We visualize samples from sample-wise perturbed CIFAR-10 unlearnable datasets in Figure 8, and from class-wise perturbed unlearnable datasets (prefixed by ◦ throughout results) in Figure 9. NTGA is omitted due to data ordering of the publicly available poison. We also visualize SVHN samples in Figure 10 and CIFAR-100 in Figure 11.

A.6 Broader Impact Statement

Our findings test prevailing hypothesis about unlearnable datasets and our results have practical implications for their use. Two of our three main conclusions relate to privacy vulnerabilities when employing unlearnable datasets for data protection. In one experiment, we demonstrate useful features can be learned from unlearnable data. In another, we demonstrate how one can effectively remove a class-wise perturbation. Our findings highlight the need for extra caution when it comes to using unlearnable datasets. By making this information available to the public, the capabilities and vulnerabilities of unlearnable datasets can be better understood.

Class O

Class 1

Class2

Class3

Class 4

Class5

Class6

Class7

Class8

Class9

OPS

Unlearnable

Examples

Regions-4

Random

Noise

Clean

Unlearnable

Examples

Adversarial

Poisoning

AR(l2)

Robust

Unlearnable

LSP

OPS+EM

Clean

OPS

Unlearnable Examples

Regions-4

Clean

Unlearnable Examples

Adversarial Poisoning

Class-wise Unlearnable Examples

Clean

Unlearnable Examples

Adversarial Poisoning

Class-wise Unlearnable Examples

1919

23

9393

616L

23

939j

23

1919

23

6161

23

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper comprehensively evaluated existing unlearnable examples and showed a surprising result. Unlearnable examples aim to prevent the model from learning useful features from the data. However, results show that several methods that networks actually can learn useful features. This is revealed by applying an existing feature reweighting method. Such results indicate that \"privacy\" might not be preserved. This paper also demonstrated that the recent findings of linear separation in unlearnable examples are not a necessary condition and an orthogonal projection attack can break class-wise linear separable unlearnable noise.\n\nSTRENGTHS:\n- By using an existing method of Deep Feature Reweighting (DFR), the finding is very interesting and new in this field. This paper has comprehensively evaluated the existing different methods for generating unlearnable examples. It is surprising to see model can \"actually\" learn useful features on several unlearnable methods. Interestingly, the error-minimizing does not. Although the exact cause remains unknown, such findings provide valuable insights for future works. Additionally, this also indicates that future works should consider DFR as a reasonable evaluation method for unlearnable examples. \n- Based on recent work, Yu et al. [34] show that unlearnable examples rely on linearly separable noise. This paper extended the analysis to a wide range of different generation methods. Results show that although unlearnable examples are commonly induced linear separable noise, one particular method, Autoregressive perturbations, does not fall into this category. This indicates that linear separable noise is not a necessary condition. These results provide insights for future works that there could be more effective non-linear separable unlearnable noise. \n- The presentations in this work are very good. It covered comprehensive related works, motivations, challenges and limitations in unlearnable examples.\n\nWEAKNESSES:\nMy main concern is the Orthogonal Projection attack. Although it is technically sound, it is unclear what is the main contribution or the insight of this proposed method. \n- Class-wise noise is known to be easily detected in unlearnable examples. Considering an additive noise, Yu et al. [34] show averaging across classes can easily expose the class-wise noise. One trivial solution that might consider is to subtract the noise for each class. Does an Orthogonal Projection attack more effective compared to this trivial solution? For Orthogonal Projection attacks, does it have any effect on the sample-wise setting?\n- Compared to adversarial training (AT), the main benefit is efficiency. However, AT has other desired properties, such as adversarial robustness and learning robust features. As for effectiveness, it seems to be no significant benefit of the Orthogonal Projection attack. For OPS or OPS+EM, the constraint is a single pixel, while AT evaluated in this paper is $L_\\infty$, changing AT constraint might be effective against the OPS-based method.\n\nQUESTIONS:\nSee the weaknesses section.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations and potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper suggests that DNNs can learn useful features from unlearnable datasets and provides a counterexample, demonstrating that linear separability of perturbations is not a necessary condition. They propose the Orthogonal Projection method to recover unlearnable datasets.\n\nSTRENGTHS:\n1. A new method called Orthogonal Projection is proposed to recover unlearnable datasets. On class-wise unlearnable datasets, this method achieves similar (or even betters) performance compared to adversarial training. Moreover, it is much faster than adversarial training.\n2. This work suggests the risk of using class-wise perturbations to craft unlearnable datasets.\n\nWEAKNESSES:\n1. For the claim \"DNNs can learn useful features from unlearnable datasets\", your experimental evidence is not sufficient and the logic is not very solid. See Question 1.\n2. Though the intuition of Orthogonal Projection is clear, the purpose of its mathematical form is still ambiguous. Especially it cannot explain why this method is good at treating class-wise perturbations rather than sample-wise ones. See Question 2.\n3. Sections 4.2, 4.3 and 4.4 have weak connections with each other, appearing as if they are independent from one another. The results in sections 4.2 and 4.3 are not interesting enough.\n\nQUESTIONS:\n1. About weakness 1:\n1.1 In Figure 2, the performance of DFR on different unlearnable datasets is not consistent. At the last epoch, DFR in (b, d) is even worse than randomly initialized baseline. For certain datsets, such as Random noise and AR, the peak in early epochs is a property of these datsets, not your DFR method.\n1.2 I don't think the increase of test accuracy caused by DFR can support you claim because you retrain the linear classifier using clean data. Don't forget that your DFR method improve the accuracy of randomly initialized baseline from $10\\%$ to $35.79\\%$. However, you can not conclude that a randomly initialized feature extractor learns useful features.\n2. About weakness 2:\n2.1 In the algorithm of Orthogonal Projection, the recovered data $X_r$ is ensured that $W^T X_r=0$. What does this equation mean? How does it work for recovery?\n2.2 What is the reason that Orthogonal Projection is not good at recovering sample-wise perturbations?\n2.3 It is mentioned in Line 262 that perturbed images of sample-wise Unlearnable Examples and Adversarial Poisoning are not linearly separable though perturbation themselves are linearly separable. Does this illustration contradict to the claim in Line 216 \"learning the most predictive features in an unlearnable dataset can amount to learning the perturbation itself\"?\n2.4 Please give experimental evidence of linear (in)separability of images perturbed by those class-wise and sample-wise attacks.\n\nLIMITATIONS:\nAs mentioned in the paper, orthogonally projection removes K dimensions from the data manifold, where K is the number of classes in a dataset. While this may not be a problem for high-resolution images with tens of thousands of dimensions, this detail could impact applicability for low-resolution datasets.\nAlso see the weakness.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper conducts an analysis of the properties of unlearnable dataset methods to evaluate their potential for future viability and security assurances. It is demonstrated that neural networks possess the ability to learn generalizable features from unlearnable datasets, while also suggesting that image privacy may not be effectively preserved. Additionally, a counterexample is provided to challenge the widely held belief that unlearnable datasets induce learning shortcuts through the linear separability of added perturbations. To address this issue, an orthogonal projection attack is proposed, which enables learning from various unlearnable datasets. The results of this approach demonstrate that linearly separable perturbations should not be relied upon.\n\nSTRENGTHS:\n1. The originality of the paper is good, as it gives us a different view to unlearnable examples and poisoning attacks by\n    (a). Demonstrating that neural networks possess the ability to learn generalizable features from unlearnable datasets \n    (b). Giving a counterexample is provided to challenge the widely held belief that unlearnable datasets induce learning shortcuts through \n      the linear separability of added perturbations\n    (c). Proposing a new evaluation framework and a novel attack method to assess the viability and security promises of unlearnable datasets.\n   Overall, The paper's findings challenge some widely held beliefs about unlearnable datasets and provide insights into their limitations and \n   potential vulnerabilities\n2. The proposed method can achieve good results on some of unlearnable tasks (class-wise) and is more effective than adversial training.\n3. This paper is well-writen\n\nWEAKNESSES:\n1. The motivation for the paper does not align to me. I hope that the proposed method is motivated by the findings (Sections 4.2 and 4.3), however, the authors only illustrate the relationship between linearly separable perturbations and the proposed orthogonal projection method, making Section 4.2 a superfluous. There should be more discussion on the relationship between the findings and the methods presented in Section 4.4.\n2. The author uses the DFR method to prove that DNN can learn useful features, but it is not convincing to me. The improvement in accuracy may simply come from using cleaner samples. (eg Figure 2, pictures (b), (c)). Also, I think the act of loss should also be involved in this part. There should be more references to why \"the higher the test accuracy after DFR, the more likely the model will pick up on private image features present in the original clean data\".\n3. Although the authors give a counterexample to challenge the commonly held view that non-learnable datasets induce learning shortcuts by adding perturbed linear separability, it lacks theoretical analysis.\n4. The method only achieves satisfactory results on Class-Wise perturbation, but the generalization of the method is not enough. In contrast, adversarial training has better generalization ability.\n5. Limited evaluation: Evaluation requires more datasets.\n\nQUESTIONS:\n1. The correlation between the method proposed in Section 4.4 and the findings in Sections 4.3 and 4.2, particularly 4.2, is of interest. \n2. Can additional sources be provided to corroborate the assertion that \"the higher the test accuracy after DFR, the more likely it is that the model has earned private image features present in the original clean data\"? \n3. The efficacy of the proposed method on other datasets merits investigation. It is recommended that further experimentation be conducted. \n4. The dissimilar trends displayed by (a) (d) and (b) (c) in Figure 2 require explanation.\n\nLIMITATIONS:\nYes, the authors adequately addressed the limitations.\nHowever, in order to further enhance and augment the credibility and validity of their work, it is suggested that they provide a more elaborate and comprehensive explanation regarding the various methodologies and motivations that were employed in their research process. Additionally, it would be highly beneficial if they could include a greater number of experiments and references within their work, as this would serve to substantially bolster and fortify the robustness and soundness of their findings and conclusions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of the actual learnability of unlearnable datasets. Specifically, the authors have demonstrated that unlearnable datasets that are generated by existing methods can actually be used to learn generalizable features. In addition, the authors show that it is not necessary to make poisons linearly separable for achieving effective poisoning effects. Furthermore, based on the fact that most existing poisoning methods rely on linear separability, the authors propose a simple yet effective countermeasure to recover clean images for training.\n\nSTRENGTHS:\n- It is valuable to revisit existing work in a popular field like data poisoning.\n- The paper is very well written, with a thorough review of related work and sufficient descriptions of technical details. \n- Experiments are extensive in terms of the number of models, datasets, settings, compared baselines, and so on. \n- Several visualizations are provided to help support new findings.\n\nWEAKNESSES:\nI very much appreciate the paper, and there are no major weaknesses but minor ones about tuning down some claims:\n\n1. This paper is not the first to find a counterexample to the linear separability hypothesis considering that previous work has intentionally relied on ideas beyond linear separability for generating unlearnable datasets [a].\n\n2. The orthogonal projection method is not the only simple yet effective countermeasure considering [b], where the simple image compression-based method, ISS, is applied to remove poisoning perturbations. About the results, it seems that ISS is more globally effective than the orthogonal projection method. In addition, It would also be appreciated if diffusion models [3] and error-maximizing augmentation [20] can be compared. These two methods are also conceptually simple because they rely on either a per-trained (diffusion) model for pre-processing or data augmentation (that may not necessarily be adversarial). \n\n[a] Is Adversarial Training Really a Silver Bullet for Mitigating Data Poisoning? ICLR 2023\n\n[b] Image Shortcut Squeezing: Countering Perturbative Availability Poisons with Compression. ICML 2023\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}