SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pedro Sandoval-Segura<sup>1</sup> Vasu Singla<sup>1</sup> Jonas Geiping<sup>1</sup>
Micah Goldblum2 Tom Goldstein1
$^{1}$ University of Maryland  $^{2}$ New York University
{psando, vsingla, jgeiping, tomg}@umd.edu goldblum@nyu.edu
In an era of widespread web scraping, unlearnable dataset methods have the potential to protect data privacy by preventing deep neural networks from generalizing. But in addition to a number of practical limitations that make their use unlikely, we make a number of findings that call into question their ability to safeguard data. First, it is widely believed that neural networks trained on unlearnable datasets only learn shortcuts, simpler rules that are not useful for generalization. In contrast, we find that networks actually can learn useful features that can be reweighed for high test performance, suggesting that image protection is not assured. Unlearnable datasets are also believed to induce learning shortcuts through linear separability of added perturbations. We provide a counterexample, demonstrating that linear separability of perturbations is not a necessary condition. To emphasize why linearly separable perturbations should not be relied upon, we propose an orthogonal projection attack which allows learning from unlearnable datasets published in ICML 2021 and ICLR 2023. Our proposed attack is significantly less complex than recently proposed techniques. $^{1}$
Deep learning is fueled by an abundance of data, the collection of which is largely unregulated [17, 20, 14, 32]. Images of human faces [9], artwork [1], and text [22, 24] are increasingly scraped at scale without consent. In an attempt to prevent the unauthorized use of data, unlearnable dataset methods make small perturbations to data so that deep neural networks (DNNs) trained on the modified data result in poor test accuracy [10, 7, 6, 36, 27]. The idea is that if generalization performance is harmed by incorporating the modified, "unlearnable" data, third parties will be disincentivized from scraping it. Unlearnable dataset methods answer the following question: How should one imperceptibly modify the clean training set to cause the largest generalization gap?
In this paper, we analyze properties of unlearnable dataset methods in order to assess their future viability and security promises. Currently, unlearnable dataset methods are unlikely to be used for safeguarding public data because they require perturbing a large portion (e.g. more than  $50\%$ ) of the training set [10, 27] and adversarial training is a principled attack [31]. Additionally, published data is immutable and must withstand current and future attacks [23]. While there are a number of reasons why these unlearnable dataset methods are unlikely to be employed, our results shed light on privacy implications and challenge common hypotheses about how they work. We make several findings by analyzing a number of unlearnable datasets developed from diverse objectives and theory. In particular,
- We demonstrate that, in many cases, neural networks can learn generalizable features from unlearnable datasets. Our results imply that while resulting test accuracy may be low, DNNs may still learn useful features from protected data.
$^{1}$ Code is available at https://github.com/psandovalsegura/learn-from-unlearnable.
Figure 1: We study unlearnable datasets constructed from additive perturbations. Although unlearnable datasets are created to prevent the exploitation of the clean dataset, we show that unlearnable datasets can be used to learn features which generalize to clean test data. We demonstrate that unlearnable datasets can be created without linearly separable perturbations. And we develop a novel method for recovering class-wise, linearly separable perturbations from unlearnable datasets.
- We challenge the common belief that unlearnable datasets work due to linearly separable perturbations. We construct a counterexample, suggesting that there is room for new unlearnable dataset methods that evade common attacks.
- Inspired by our results on linear separability of perturbations, we present a new Orthogonal Projection attack which allows learning from unlearnable datasets perturbed by class-wise, linearly separable perturbations. Our attack demonstrates that class-wise applied perturbations do not protect data from being learned. Our attack is often more effective than adversarial training, at a fraction of the computational cost.
Terminology Data poisoning methods that perturb the entire training dataset, and which we refer to as "unlearnable datasets" or simply "poisons", are also known as availability attacks [7, 36], generalization attacks [37], delusive attacks [31], or simply unlearnable examples [35]. Throughout this work, unlearnable datasets are considered defenses, given that their primary use case is to prevent the exploitation of data. Aiming to learn from unlearnable datasets is considered an attack.
Unlearnable Datasets One of the earliest instances of a data poisoning attack intended to reduce overall test performance is from [2], who optimize a single sample that corrupts SVM training. Poisoning a convex model like SVM or logistic regression can be solved exactly, but poisoning deep neural networks is more challenging. A number of approaches have been proposed, validated primarily by empirical evidence due to the complexity of the objective (See Eq. 1 and 2). [5] propose using an auto-encoder-like network to generate perturbations against a victim classifier, [37] use NTKs to understand network predictions and optimize perturbations to produce misclassifications, [36] generate linearly separable perturbations, [10, 28] optimize error-minimizing noise whereas [7] optimize error-maximizing noise, and [27] generate autoregressive patterns, among other methods. The diversity of approaches motivates us to understand whether these unlearnable datasets share any properties in common, as we explore in Section 4.3.
Privacy Vulnerabilities In the context of image classification, unlearnable datasets are said to protect privacy by ensuring the modified data is not used or included as part of a larger dataset. That is, training on only poisoned data should prevent test set generalization and training on a dataset consisting of clean and poisoned data should be no better than training just on the clean portion. If poisoned training data is rendered useless for test set generalization, the data will not be used. In this way, only the initial owner can train on the original, unperturbed data and achieve high generalization performance. In recent work, [19] introduce the ability to "lock" and "unlock" training data by leveraging a class-wise perturbation, removable only by someone with knowledge of the exact perturbation. Unfortunately, as we empirically demonstrate in Section 4.4, class-wise perturbations can give a false sense of security. An argument against the use of poisoning for data privacy is the unavoidable reality that published data is immutable, and thus must withstand current and future methods which may exploit the data [23]. While it remains possible to recover relatively high test
accuracy from unlearnable datasets, our experiments also suggest that there is not yet a way to restore test accuracy to levels seen during clean training. Unlearnable datasets are known to be vulnerable to adversarial training [10, 31]. By formalizing unlearnable datasets as finding the worst-case training data within a  $\infty$ -Wasserstein ball, [31] find that adversarial training [16] is a principled defense and can recover a high degree of accuracy by preventing models from relying on non-robust features. [28] challenge this assertion and optimize perturbations designed to degrade model performance under adversarial training. In Section 4.4, we show that adversarial training remains a strong defense against unlearnable datasets. In fact, we find that hyperparameter changes to the adversarial training recipe allows learning from Robust Unlearnable Examples [28], demonstrating the brittleness of unlearnable datasets.
How Unlearnable Datasets Work There are many explanations for how unlearnable datasets prevent networks from test set generalization: error-minimizing perturbations cause overfitting [10], error-maximizing noise stimulates learning non-robust features [7], convolutional layers are receptive to autoregressive patterns [27], and more. There are a variety of explanations because different methods arose from different optimization objectives and theory. But the leading explanation comes from Yu et al. [36], who find near perfect linear separability of perturbations for all the unlearnable datasets they consider. They explain that unlearnable datasets cause learning shortcuts due to linear separability of perturbations. In Section 4.3, we find a counterexample, demonstrating that while linear separability of perturbations may certainly be a property that helps unlearnable datasets function, the property is not necessary.
We consider the problem of creating a clean-label unlearnable dataset in the context of a  $K$ -way image classification task, following [10]. We denote the clean training and test datasets as  $\mathcal{D}_{\mathrm{train}}$  and  $\mathcal{D}_{\mathrm{test}}$ , respectively.
Suppose there are  $n$  samples in a clean training set, i.e.  $\mathcal{D}_{\mathrm{train}} = \{(x_i,y_i)\}_{i=1}^n$  where  $x_i \in \mathbb{R}^d$  are the inputs and  $y_i \in \{1,\dots,K\}$  are the labels. We consider a unlearnable dataset, denoted  $\widetilde{\mathcal{D}}_{\mathrm{train}} = \{(x_i',y_i)\}_{i=1}^n$  where  $x_i' = x_i + \delta_i$  is the perturbed or poisoned version of the example  $x_i \in \mathcal{D}_{\mathrm{train}}$  and where  $\delta_i \in \Delta \subset \mathbb{R}^d$  is the perturbation. The set of allowable perturbations,  $\Delta$ , is typically an  $\ell_p$ -ball with radius  $\epsilon$  where  $\epsilon$  is small enough that  $\delta$  is imperceptible.
Unlearnable datasets are created by applying a perturbation to a clean image in either a class-wise or sample-wise manner. When a perturbation is applied class-wise, every sample of a given class is perturbed in the same way. That is,  $x_{i}^{\prime} = x_{i} + \delta_{y_{i}}$  and  $\delta_{y_i}\in \Delta_C = \{\delta_1,\dots,\delta_K\}$ . When samples are perturbed in a sample-wise manner, every sample has a unique perturbation.
All unlearnable dataset methods aim to solve the following bi-level maximization:
$$
\max  _ {\delta \in \Delta} \mathbb {E} _ {(x, y) \sim \mathcal {D} _ {\text {t e s t}}} [ \mathcal {L} (f (x), y; \theta (\delta)) ] \tag {1}
$$
$$
\theta (\delta) = \underset {\theta} {\arg \min } \mathbb {E} _ {\left(x _ {i}, y _ {i}\right) \sim \mathcal {D} _ {\text {t r a i n}}} [ \mathcal {L} (f \left(x _ {i} + \delta_ {i}\right), y _ {i}; \theta) ] \tag {2}
$$
Eq. 2 describes the process of training a model on unlearnable data, where  $\theta$  denotes the model parameters. Eq. 1 states that the unlearnable data should be chosen so that the trained network has high test loss, and thus fails to generalize to the test set.
4.1 Datasets and Training Settings
For all of our experiments, we make use of open-source unlearnable datasets: From Unlearnable Examples [10], we use their sample-wise and class-wise poisons. From Adversarial Poisoning [7], we use their targeted PGD attack poison. From Autoregressive Poisoning (AR) [27], Neural Tangent Generalization Attacks (NTGA) [37], Robust Unlearnable Examples [28], and Linearly Separable
Table 1: Generalizable features can be learned from unlearnable datasets. While published unlearnable datasets cause DNNs to train to low test accuracy, the features learned by DNNs can be reweighted using DFR to high test accuracy in many cases. For each unlearnable dataset, we report test accuracy and test loss for the best performing checkpoint after DFR. In gray, we indicate test accuracy improvement/deterioration over DFR on a randomly initialized RN-18. DFR uses 5,000 clean samples for finetuning. For the majority of unlearnable datasets, peak performance seems to occur early in training. High DFR Test Accuracy, and Low DFR Test Loss, indicates that useful features are learned during training.
TRAINING DATA	MAX DFR TEST ACCURACY	MIN DFR TEST LOSS
NONE	35.97	2.379
UNLEARNABLE EXAMPLES [10]	39.56 (+3.59)	1.798
ADVERSARIAL POISONING [7]	69.99 (+34.02)	1.036
AR (l2) [27]	58.73 (+22.76)	1.531
NTGA [37]	57.12 (+21.15)	1.391
ROBUST UNLEARNABLE [28]	41.02 (+5.05)	1.790
LSP [36]	43.31 (+7.34)	1.675
OPS+EM [35]	38.98 (+3.01)	1.869
○OPS [35]	47.70 (+11.73)	1.697
○UNLEARNABLE EXAMPLES [10]	34.09 (-1.88)	2.037
○REGIONS-4 [26]	48.09 (+12.12)	1.706
○RANDOM NOISE	66.79 (+30.82)	1.352
Perturbations (LSP) [36], we use their main poison. For One-pixel Shortcut [35], we use their OPS and CIFAR-10-S poisons, but we refer to their CIFAR-10-S poison as  $OPS + EM$  for ease of attribution. We also include two class-wise Regions-4 and Random Noise poisons which contain randomly sampled perturbations, following [26]. To poison a  $K$ -class dataset in a class-wise manner, we need  $K$  perturbation vectors. For Regions-4, each perturbation is independently created by sampling 4 vectors of size 3 from a Bernoulli distribution. Each vector is then scaled to lie in the range  $\left[-\frac{8}{255}, \frac{8}{255}\right]$ . Finally, each of the 4 vectors are repeated along height and width dimensions to create patches of size  $16 \times 16$ , which are then arranged side by side to achieve a shape of  $32 \times 32$ . For Random Noise perturbations, we sample an i.i.d. vector, one entry per pixel, from a Bernoulli and scale perturbations to fit the imperceptibility constraint. See Appendix A.5 for image samples for all unlearnable datasets we consider.
Results in this section are for CIFAR-10 [13], and additional results for SVHN [18], CIFAR-100 [13], and ImageNet [25] Subset are in Appendix A.2.2 and A.4.2. Unlearnable datasets constructed with class-wise perturbations are prefixed with  $\circ$ , otherwise the perturbations were added sample-wise. For the imperceptibility constraint, the  $AR(\ell_2)$  dataset contains perturbations  $\delta$  of size  $\|\delta\|_2 \leq 1$ .  $LSP$  has perturbations of size  $\|\delta\|_2 \leq \epsilon' \sqrt{d}$ , where  $d$  is the image dimension and  $\epsilon' = \frac{6}{255}$ .  $OPS$  and  $OPS+EM$  contain unbounded perturbations. All other datasets contain perturbations of size  $\|\delta\|_\infty \leq \frac{8}{255}$ .
We primarily use the ResNet-18 (RN-18) [8] architecture for training on the above datasets. We include results for additional network architectures including VGG-16 [29], GoogleNet [30], and ViT [4] in Appendix A.2.1 and A.4.1. Training hyperparameters can be found in Appendix A.1.
4.2 DNNs Can Learn Useful Features From Unlearnable Datasets
Unlearnable datasets are meant to be unlearnable; models trained on unlearnable datasets should not be capable of generalizing to clean test data. Presumably, the modified, unlearnable dataset protects the clean version of the dataset from unauthorized use in this fashion. But by reweighting deep features, we find that DNNs are still able to recover some generalizable features from unlearnable datasets.
DFR Method The technique of training a new linear classification layer on top of a fixed feature extractor is known as Deep Feature Reweighting (DFR) [12] and it is a method for learning from data with spurious correlations, where predictive features in the train distribution are not present in the test
Figure 2: Features learned during poison training can be reweighed for high test accuracy. (a) When training on Adversarial Poisoning, RN-18 test accuracy (solid line) drops below random chance test accuracy. For the dashed line, each data point at epoch  $i$  represents the final test accuracy obtained after DFR using the poisoned checkpoint from epoch  $i$  of standard training. Using DFR, the final RN-18 poisoned checkpoint can achieve nearly  $70\%$  test accuracy (dashed line). Using DFR on a randomly initialized RN-18 only achieves  $40\%$  test accuracy (dotted line). (b-d) For different unlearnable datasets, at different points during poison training (marked by star), DFR test accuracy is more than  $20\%$  above the accuracy of a randomly initialized feature extractor, suggesting that features learned from unlearnable data are relevant for clean test set generalization.
distribution. We borrow the DFR method to better understand the utility of features learned during poison training. The higher the test accuracy after DFR, the more likely it is that the model has learned features present in the original clean data. If DFR works well on poison-trained networks, this would suggest that even poisoned weights contain a semblance of relevant information for generalization.
To evaluate the extent that generalizable features are learned, we start by saving network weights at every epoch of poison training. In the context of image classification, network weights consist of a feature extractor followed by a fully-connected classification layer mapping feature vectors to class logits. Next, for each checkpoint, we utilize a random subset of 5,000 clean CIFAR-10 training samples (10% of the original training set) to train a new classification head (and keep the feature extractor weights fixed). Finally, we plot test accuracy in Figure 2 and evaluate the maximum CIFAR-10 test accuracy achieved through DFR for each dataset and report results in Table 1. As a baseline, we train a classification head on a randomly initialized RN-18 feature extractor and find that these random features can be reweighted to achieve 35.97% test accuracy.
DFR Results We find that, to different extents, DNNs do learn generalizable features from unlearnable datasets. Surprisingly, on Adversarial Poisoning and NTGA, RN-18 features actually improve throughout training, as we show in Figure 2 (a) and (d), despite low test accuracy during poison training (solid lines in Figure 2 (a) and (d)). On AR and  $\circ$  Random Noise, test accuracy peaks early in training before dropping to random chance accuracy (solid lines in Figure 2 (b) and (c)). Still, when early checkpoints are used, one can use features from the poison-trained feature extractors
Table 2: Linearly separable perturbations are not necessary to create unlearnable datasets. We train a linear logistic regression model on perturbations and report train accuracy. High train accuracy indicates linear separability of perturbations. Unlike perturbations from other unlearnable datasets, Autoregressive Perturbations (AR) are not linearly separable and are, in fact, less separable than clean CIFAR-10 images.
TRAINING DATA	TRAIN ACCURACY
CLEAN	53.88
UNLEARNABLE EXAMPLES [10]	100
ADVERSARIAL POISONING [7]	100
AR (l2) [27]	39.58
NTGA [37]	100
ROBUST UNLEARNABLE [28]	100
LSP [36]	100
OPS+EM [35]	100
○OPS [35]	99.93
○UNLEARNABLE EXAMPLES [10]	100
○REGIONS-4 [26]	100
○RANDOM NOISE	100
to achieve high accuracy (dashed lines in Figure 2 (b) and (c)). Accuracy curves for Adversarial Poisoning are unique because images are perturbed with error-maximizing noise, which correspond to actual features models use during classification [11]. In this case, DFR is reweighting useful, existing features for classification, leading to higher test accuracy. On the other hand, Random Noise and AR poisons do not perturb images with useful features; instead, both perturb with synthetic noise. In these cases, useful features are still learned during poison training, but only in the first epochs. As training progresses, the model checkpoints are continually corrupted by synthetic noise features which cannot be useful for classification despite reweighting.
As we show in Table 1,  $\circ$  Unlearnable Examples are very effective at corrupting network weights during training; using DFR on checkpoints from poison training only yields a maximum test accuracy of  $34.09\%$ , nearly  $2\%$  worse than a randomly initialized feature extractor. Six of the eleven unlearnable datasets we consider yield checkpoints which achieve DFR test accuracies that are  $10\%$  or more higher than a randomly initialized feature extractor. Generally, it is interesting that unlearnable dataset methods with the lowest max DFR test accuracy are those which use error-minimizing perturbations in some capacity, e.g. Unlearnable Examples, Robust Unlearnable Examples, and  $OPS + EM$ . Analyzing test loss follows trends from test accuracy: Adversarial Poisoning, AR, NTGA, and Random Noise achieve lowest losses – and those poisoned checkpoints also have the highest DFR test accuracy in Table 1. More interestingly, we find that all poisoned models have a lower loss than the randomly initialized model. This reinforces our claim that models learned useful features from poisoned data.
Overall, our results suggest that network weights during poison training are not fully corrupted in many cases. While poison-trained networks may be evaluated to have low test accuracy, we show that, for some unlearnable datasets, the networks have learned generalizable features which can be reweighted for high test performance. It is entirely possible that these features, which encode the original data amount to useful image information that the original data owner did not want incorporated into an ML system.
Privacy concerns are often cited as primary motivation for unlearnable datasets, yet our results raise concerns about this framing. A model trained on an apparently unlearnable dataset might have low test error, but this does not imply, as we show, that it does not contain usable information about the original data, and might not keep promises to protect data privacy.
4.3 Linearly Separable Perturbations Are Not Necessary
The work of Yu et al. [36] demonstrated that unlearnable datasets contain linearly separable perturbations and suggested that linear separability is a necessary property of unlearnable datasets. However,
2On 10 class classification task, the expected random chance loss is  $-\ln \left(\frac{1}{10}\right) \approx 2.302$
Algorithm 1 Orthogonal Projection
Input: Unlearnable Dataset,  $(X,Y)\in \widetilde{\mathcal{D}}_{\mathrm{train}}$
Output: Recovered Dataset,  $(X_r,Y)$
1: while not converged do
2: Sample batch  $(x,y)$  from  $\widetilde{\mathcal{D}}_{\mathrm{train}}$
3:  $W\gets W - \eta \nabla_W L(W^T x,y)$
4: end while
5: Perform QR decomposition on  $W$  to obtain  $Q$  matrix
6:  $X_{r}\gets X - QQ^{T}X$
by finding a counterexample, we confirm that linear separability is not necessary to have an effective unlearnable dataset.
Following [36], we train linear logistic regression models on image perturbations by subtracting the clean image from the perturbed image. Given the publication of new unlearnable datasets, we add to Yu et al.'s analysis by including AR, Robust Unlearnable Examples, and OPS poisons. It is possible to obtain image perturbations given our access to clean CIFAR-10, but should not be possible in practice if the clean version of the unlearnable dataset is unavailable to the public. We optimize the logistic regression weights  $(3072 \times 10$  parameters for CIFAR-10) using L-BFGS [15] for 500 steps using a learning rate of 0.5. For reference, we also train a logistic regression model on clean CIFAR-10 image pixels and report the train accuracy in the first row of Table 2. For results on the linear separability of the perturbed images, see Appendix A.3.2.