SlowGuess/NeurIPS_2023_OCR huggingface dataset:
While most current unlearnable datasets contain linearly separable perturbations, as was initially shown by Yu et al. [36],  $AR$  perturbations stand out as a counterexample in Table 2. After training on  $AR$  perturbations, a logistic regression model can only achieve  $39.58\%$  train accuracy while all other unlearnable datasets we tested contain perturbations that are almost perfectly linearly separable. Because most unlearnable dataset methods developed from diverse optimization objectives result in linearly separable perturbations, we posit that linear separability is an easy solution for optimization of Eq. 1 and 2 to find. For unlearnable datasets that are not optimized, class-wise perturbations are trivially linearly separable, so long as no two class perturbations are the same. Thus, although  $AR$  perturbations are not linearly separable, they remain separable by a simple 2-layer CNN [27], suggesting that simple perturbations may be a more accurate way of defining effective unlearnable perturbations.
Being the first counterexample of its kind, we believe there may be underexplored unlearnable dataset methods. This finding sheds light on the complexity of unlearnable datasets, whose behavior depends on choices of loss function, optimizer, and even network architecture. An important detail about  $AR$  perturbations is that they are not optimized – they are generated [27]. We leave to future work an investigation as to why most optimized perturbations in unlearnable datasets to date result in linear separability.
4.4 Orthogonal Projection for Learning From Datasets with Class-wise, Linear Perturbations
In this section, we develop a method to learn from unlearnable datasets. Given our understanding of linear separability from Section 4.3, learning the most predictive features in an unlearnable dataset can amount to learning the perturbation itself. We leverage this intuition in our Orthogonal Projection attack, which works by learning a linear model, then projecting the data to be orthogonal to linear weights. In other words, we remove the most predictive dimensions of the data. For unlearnable datasets perturbed by class-wise, linearly separable perturbations, the most predictive dimensions are easily found by a linear model. Attacks for learning from class-wise perturbed unlearnable datasets have included using diffusion models [3], adversarial training [10], and error-maximizing augmentations [21]. In contrast to these techniques, our method is simpler and computationally cheaper.
4.4.1 Orthogonal Projection Method
Unlearnable datasets with class-wise perturbations are so simple that visualizing the average image of a class can expose the class perturbation (see Figure 5). In order to adaptively learn these perturbations and remove them from poisoned data, we propose a method that first learns the simple perturbations and then orthogonally projects samples to omit them. Our Orthogonal Projection method is designed
(b)
Figure 3: Our Orthogonal Projection attack learns linearly separable features. In (a), we visualize how LSP [36] and OPS [35] unlearnable data is created by taking clean image and adding a perturbation. In (b), we visualize learned weights ( $W$  in Algorithm 1) after training a linear model on LSP and OPS images; we also include learned weights after training on clean CIFAR-10 for reference. Learned weights from LSP and OPS unlearnable data nearly match the original perturbation, while learned weights from clean data resemble the corresponding class. In our Orthogonal Projection attack, we project each perturbed image to be orthogonal to each of these learned weights (Algorithm 1, Line 6).
to exploit class-wise perturbations by design and is meant to emphasize why class-wise perturbations should not be relied on for creating unlearnable datasets.
Given an unlearnable dataset, we train a simple logistic regression model on poison image pixels, in an attempt to optimize the most predictive linear image features. The result is a learned feature matrix  $W$ , on which we perform a QR decomposition such that  $W = QR$ , where  $Q$  consists of orthonormal columns. Image samples  $X$  from the unlearnable dataset can be orthogonally projected using the matrix transformation  $I - QQ^T$ . Orthogonal projection ensures that the dot product of a row of  $X$  with every column of  $Q$  is zero, so that learned perturbations from the logistic regression model are not seen during the final training run on recovered data. The recovered image samples are then written as  $X_r = (I - QQ^T)X$ . Finally, we train the final network on recovered data,  $X_r$ . A detailed algorithm is shown in Algorithm 1. We include additional intuition in Appendix A.4.3.
The added computational cost of Orthogonal Projection over standard training is only in training the logistic regression model and projecting the dataset, which can be done once. For CIFAR-10, we train the logistic regression model for 20 epochs using SGD with an initial learning rate of 0.1, which decays by a factor of 10 on epochs 5 and 8. We find that training the linear model for longer is often detrimental, potentially because of overfitting.
4.4.2 Orthogonal Projection Results
We compare our attack against  $\ell_{\infty}$  adversarial training, as previous work has shown that adversarial training is a principled defense [31] against unlearnable datasets. For adversarial training, we use a 3-step PGD adversary with a perturbation radius of  $\frac{8}{255}$  in  $\ell_{\infty}$ -norm. Note that large adversarial training perturbation radii harm clean accuracy [33]. We use SGD with momentum of 0.9 and an initial learning rate of 0.1, which decays by a factor of 10 on epoch 75, 90, and 100.
We find our method is more effective than adversarial training when learning from unlearnable datasets corrupted in a class-wise manner. In Table 3, our Orthogonal Projection attack leads to greater test accuracy on all class-wise poisons prefixed by  $\circ$ . By visualizing the learned weights from
Table 3: Orthogonal Projection can make class-wise unlearnable data learnable. Especially for unlearnable datasets with class-wise, linearly separable perturbations, our Orthogonal Projection attack improves CIFAR-10 test accuracy over  $\ell_{\infty}$  Adversarial Training at a fraction of the computational cost. We train RN-18 using different kinds of unlearnable training data and, for every attack, we indicate accuracy improvement/deterioration over standard training (no attack) in gray. For every dataset, we bold the highest test accuracy achieved.
TRAINING DATA	ATTACK
ADV TRAINING	ORTHO PROJ (OURS)
CLEAN	94.37	87.16 (-7.21)	90.16 (-4.21)
UNLEARNABLE EXAMPLES [10]	24.56	85.64 (+61.08)	65.17 (+40.61)
ADVERSARIAL POISONING [7]	7.96	85.32 (+77.36)	14.74 (+6.78)
AR (l2) [27]	13.77	84.09 (+70.32)	13.03 (-0.74)
NTGA [37]	40.78	84.85 (+44.07)	82.21 (+41.43)
ROBUST UNLEARNABLE [28]	26.86	87.07 (+60.21)	25.83 (-1.03)
LSP [36]	25.75	86.18 (+60.43)	87.99 (+62.24)
OPS+EM [35]	20.11	12.22 (-7.89)	39.43 (+19.32)
○OPS [35]	15.35	11.77 (-3.58)	87.94 (+72.59)
○UNLEARNABLE EXAMPLES [10]	14.49	85.56 (+71.07)	89.98 (+75.49)
○REGIONS-4 [26]	10.32	85.74 (+75.42)	86.87 (+76.55)
○RANDOM NOISE	10.03	86.36 (+76.33)	90.37 (+80.34)
the linear model in Figure 3, we see that the class-wise, linearly separable perturbations are recovered. Projecting orthogonal to these vectors effectively removes those features from the unlearnable dataset, allowing training to proceed without predictive, but semantically meaningless features. It should not be surprising that Orthogonal Projection is also effective against  $OPS$  (ICLR 2023), given that all images of a class have the same pixel modified. In Figure 3, we visualize the linear model weights and show that it is able to pinpoint the modified class pixel, whose influence is effectively removed after projection in our method. We visualize additional linear model weights for other datasets in Appendix A.4.5. Orthogonal Projection is also surprisingly effective against NTGA (ICML 2021), such that an RN-18 model can achieve  $82.21\%$  test accuracy when trained on orthogonally projected NTGA images. For seven of the eleven unlearnable datasets we consider, Orthogonal Projection can achieve above  $80\%$  test accuracy without adversarial training. Unlike the linear model from Section 4.3 which is trained on only perturbations, the linear model that we train for Orthogonal Projection trains on perturbed images and the resulting data is no longer linearly separable. For this reason, Orthogonal Projection struggles against Adversarial Poisoning, AR, and Unlearnable Examples; the training distribution of the linear model is more complex than just perturbations. Given that  $OPS + EM$  is a combination of  $OPS$  and class-wise error-minimizing noise from Unlearnable Examples, both Orthogonal Projection and adversarial training are ineffective. For both  $OPS$  and  $OPS + EM$ , adversarial training achieves approximately  $12\%$  accuracy, a dismal result that is expected given the unbounded  $OPS$  perturbations. A limitation of our approach is that, by orthogonally projecting data, we effectively remove  $K$  dimensions from the data manifold, where  $K$  is the number of classes in a dataset. While this may not be a problem for high-resolution images with tens of thousands of dimensions, this detail could impact applicability for low-resolution datasets.
The susceptibility of using class-wise perturbations to craft unlearnable datasets should be expected, given that every image of a class contains the same image feature that is perfectly predictive of the label. Class-wise unlearnable datasets contain simple features that are perfectly correlated with the label and the linear logistic regression model is able to optimize features resembling the original perturbations. By design, our orthogonal projection attack is most effective against class-wise perturbed unlearnable datasets. It serves as evidence that class-wise, linearly separable perturbations cannot be relied upon for protecting data.
We design experiments to test prevailing hypotheses about unlearnable datasets, and our results have practical implications for their use. First, while all unlearnable datasets cause low test accuracy for
trained models, we demonstrate that generalizable features can still be learned from data. In many cases, high test accuracy can be achieved from a feature extractor trained on unlearnable datasets. If data were modified for protection, an adversary may still be able to train a reasonable feature extractor, a harm for data thought to be secure. We advocate for our evaluation framework to be used for auditing whether new unlearnable datasets prevent useful features from being learned. Second, we find that the reason unlearnable datasets cause low test accuracy is not as simple as previously thought. AR perturbations serve as a counterexample to the hypothesis that unlearnable datasets contain linearly separable perturbations. Our finding that AR perturbations are not linearly separable suggests there could be underexplored methods for protecting data using imperceptible perturbations. Although unlearnable datasets need not have linearly separable perturbations, if they do have them applied in a class-wise manner, we present an Orthogonal Projection attack that can effectively remove them. While learning the perturbations from perturbed images is challenging, our results imply class-wise perturbations cannot not be relied upon for data protection. We can learn a significant amount from unlearnable datasets. DNNs can learn generalizable features from unlearnable datasets assumed to be protected, linear separability is not a necessary condition for preventing generalization from unlearnable datasets, and class-wise perturbations can be optimized against and effectively removed.
This material is based upon work supported by the National Science Foundation under Grant No. IIS-1910132 and Grant No. IIS-2212182, and by DARPA's Guaranteeing AI Robustness Against Deception (GARD) program under #HR00112020007. Tom Goldstein was supported by the ONR MURI program, the AFOSR MURI program, and the National Science Foundation (IIS-2212182 & 2229885). Vasu Singla was supported in part by the National Science Foundation under grant number IIS-2213335. Pedro Sandoval-Segura is supported by a National Defense Science and Engineering Graduate (NDSEG) Fellowship. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
[1] Andy Baio. Exploring 12 million of the 2.3 billion images used to train stable diffusion's image generator, Sep 2022.
[2] Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. In Proceedings of the 29th International Coference on International Conference on Machine Learning, ICML'12, page 1467-1474, Madison, WI, USA, 2012. Omnipress.
[3] Hadi M. Dolatabadi, Sarah Erfani, and Christopher Leckie. The devil's advocate: Shattering the illusion of unexploitable data using diffusion models, 2023.
[4] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
[5] Ji Feng, Qi-Zhi Cai, and Zhi-Hua Zhou. Learning to confuse: generating training time adversarial data with auto-encoder. Advances in Neural Information Processing Systems, 32, 2019.
[6] Liam Fowl, Ping-yeh Chiang, Micah Goldblum, Jonas Geiping, Arpit Bansal, Wojtek Czaja, and Tom Goldstein. Preventing unauthorized use of proprietary data: Poisoning for secure dataset release. arXiv preprint arXiv:2103.02683, 2021.
[7] Liam Fowl, Micah Goldblum, Ping-yeh Chiang, Jonas Geiping, Wojciech Czaja, and Tom Goldstein. Adversarial examples make strong poisons. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 30339-30351. Curran Associates, Inc., 2021.
[8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[9] Kashmir Hill. The secretive company that might end privacy as we know it, Jan 2020.
[10] Hanxun Huang, Xingjun Ma, Sarah Monazam Erfani, James Bailey, and Yisen Wang. Unlearnable examples: Making personal data unexploitable. In International Conference on Learning Representations, 2021.
[34] Rui Wen, Zhengyu Zhao, Zhuoran Liu, Michael Backes, Tianhao Wang, and Yang Zhang. Is adversarial training really a silver bullet for mitigating data poisoning? In The Eleventh International Conference on Learning Representations, 2023.
[35] Shutong Wu, Sizhe Chen, Cihang Xie, and Xiaolin Huang. One-pixel shortcut: On the learning preference of deep neural networks. In The Eleventh International Conference on Learning Representations, 2023.
[36] Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie-Yan Liu. Availability attacks create shortcuts. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. ACM, aug 2022.
[37] Chia-Hung Yuan and Shan-Hung Wu. Neural tangent generalization attacks. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12230-12240. PMLR, 18-24 Jul 2021.
A.1 Training Hyperparameters
DNNs Can Learn Useful Features From Unlearnable Datasets In Section 4.2, we train a number of ResNet-18 (RN-18) [8] models on different unlearnable datasets with cross-entropy loss for 60 epochs using a batch size of 128. We save checkpoints at every epoch of training. For our optimizer, we use SGD with momentum of 0.9 and weight decay of  $5 \times 10^{-4}$ . We use an initial learning rate of 0.1 which decays using a cosine annealing schedule.
For training a new classification layer on feature extractor checkpoints, we use 5,000 random clean images from the original training set data. Note that 5,000 images is  $10\%$  of CIFAR-10 and CIFAR-100, but  $3.9\%$  of Imagenet subset, etc. Following [12], using the feature extractor, we extract embeddings from this clean subset of data and preprocess the embeddings to have mean zero and unit standard deviation. To retrain the last layer, we use the logistic regression implementation from scikit-learn (sklearn.linear_model.LogisticRegression).
Linearly Separable Perturbations Are Not Necessary In Section 4.3, for every unlearnable dataset, we first gather the set of perturbations by subtracting the clean image from the perturbed image. We zero-one normalize the perturbations before training a linear layer using L-BFGS [15] for 500 steps using a learning rate of 0.5.
Orthogonal Projection for Learning From Datasets with Class-wise, Linear Perturbations In Section 4.4, for CIFAR-10, we train the logistic regression model for 20 epochs using SGD with an initial learning rate of 0.1, which decays by a factor of 10 on epochs 5 and 8 (at epochs which are 0.5 and 0.75 through training). For CIFAR-100, we train the logistic regression model for 60 epochs due to the higher number of classes. After we orthogonally project the unlearnable data using the optimized weights, we train networks using the hyperparameters from checkpoint-training of Section 4.2.
A.2 Additional Section 4.2 Results: DNNs Can Learn Useful Features From Unlearnable Datasets
A.2.1 More Model Architectures for Section 4.2
We consider three more model architectures: VGG-16 [29], GoogLeNet [30], and ViT [4]. Our ViT uses a patch size of 4. For RN-18 and VGG-16, feature vectors are 512-dimensional. Feature vectors for GoogleNet are 1024-dimensional. The ViT class token is 384-dimensional.
Table 4: Generalizable features can be learned from unlearnable datasets, using a variety of network architectures. We report Max DFR Test Accuracy for each CIFAR-10 unlearnable dataset. In gray, we indicate test accuracy improvement/deterioration over DFR on the corresponding randomly initialized model architecture.
CIFAR-10 TRAINING DATA	MODEL ARCHITECTURE
VGG-16	GOOGENET	ViT
NONE	35.69	48.08	37.40
UNLEARNABLE EXAMPLES [10]	37.84 (+2.15)	41.08 (-7.00)	49.57 (+12.17)
ADVERSARIAL POISONING [7]	64.73 (+29.04)	71.70 (+23.62)	68.97 (+31.57)
AR (l2) [27]	36.98 (+1.29)	40.12 (-7.96)	60.53 (+23.13)
NTGA [37]	56.03 (+20.34)	61.24 (+13.16)	60.53 (+23.13)
ROBUST UNLEARNABLE [28]	39.13 (+3.44)	40.59 (-7.49)	49.10 (+11.70)
LSP [36]	40.86 (+5.17)	58.22 (+10.14)	50.95 (+13.55)
OPS+EM [35]	31.31 (-4.38)	38.57 (-9.51)	49.73 (+12.33)
○OPS [35]	39.63 (+3.94)	52.02 (+3.94)	56.04 (+18.64)
○UNLEARNABLE EXAMPLES [10]	30.47 (-5.22)	36.32 (-11.76)	44.90 (+7.50)
○REGIONS-4 [26]	43.29 (+7.60)	48.65 (+0.57)	52.60 (+15.20)
○RANDOM NOISE	72.08 (+36.39)	62.19 (+14.11)	55.58 (+18.18)
In Table 4, we find that across architectures, Adversarial Poisoning data is easiest to extract generalizable features from. Surprisingly, ViT is most effective at learning generalizable features from all unlearnable datasets, achieving more than  $7\%$  test accuracy improvement over a randomly initialized ViT in all cases. For example, using only 5,000 clean CIFAR-10 samples can be used to achieve nearly  $69\%$  test accuracy, while using the same clean samples can only achieve  $37.40\%$  test accuracy on a randomly initialized ViT. The GoogleNet architecture weights are seemingly more easily corrupted during training; Max DFR Test Accuracy for Unlearnable Examples, AR, Robust Unlearnable, and other datasets is much lower than test accuracy from a finetuned randomly initialized GoogleNet. Interestingly, the randomly initialized GoogleNet feature extractor achieves the highest DFR test accuracy.
A.2.2 More Datasets for Section 4.2
We consider three additional base datasets for four unlearnable dataset methods. We use an ImageNet [25] subset of the first 100 classes, following [10]. The train split consists of 129, 395 images, while the test split consists of 5,000 images.
Our SVHN [18], CIFAR100 [13], and Adversarial Poisoning ImageNet subset datasets contain perturbations of size  $\| \delta \|_{\infty} \leq \frac{8}{255}$ . Unlearnable Examples ImageNet [25] subset contains perturbations of size  $\| \delta \|_{\infty} \leq \frac{16}{255}$ , following their open-source repository. We generate the Adversarial Poisoning [7] ImageNet subset from published source code using 1 PGD restart, as opposed to 8 due to computation time. Our SVHN and CIFAR-100 Adversarial Poisoning datasets use 3 PGD restarts. On clean SVHN, CIFAR-100, and ImageNet subset, RN-18 achieves  $96.33\%$ ,  $74.14\%$ ,  $78.92\%$  test accuracy respectively.
Table 5: Generalizable features can be learned from unlearnable datasets of different underlying distributions. We report Max DFR Test Accuracy for each unlearnable dataset. RN-18 checkpoints are trained on SVHN, CIFAR-100, and ImageNet subset unlearnable datasets, and DFR is performed using 5,000 clean samples from the corresponding base dataset.
TRAINING DATA	SVHN	FINETUNE DATA
CIFAR-100	IMAGENET
NONE	32.05	8.13	3.64
UNLEARNABLE EXAMPLES [10]	26.76	16.12	8.44
ADVERSARIAL POISONING [7]	87.06	44.37	20.22
○ UNLEARNABLE EXAMPLES [10]	22.48	10.32	7.88
○ RANDOM NOISE	27.35	47.41	23.30
In Table 5, we again show that Adversarial Poisoning unlearnable data can be easily used to extract generalizable features regardless of the underlying distribution (base dataset of SVHN, CIFAR-100, or ImageNet). For SVHN, Adversarial Poisoning is the only dataset from which the trained feature extractor performs better in Max DFR Test Accuracy  $(87.06\%)$  over a randomly initialized RN-18  $(32.05\%)$ . As mentioned in Section 4.2, error-minimizing perturbations of Unlearnable Examples tend to be most effective at corrupting weights during training, regardless of underlying finetune data.
A.2.3 Additional Plots for Section 4.2
We add results to the experiment from Figure 2. In Figure 4, unlearnable datasets sufficiently corrupt RN-18 weights during training and prevent DFR from recovering test accuracy.
Figure 4: Representations learned by other poisons we consider are no better than random. (a-b) Reweighting deep features from sample-wise error minimizing noises provide no benefit over random features. DFR on a randomly initialized RN-18 only achieves  $40\%$  test accuracy (dotted line). (c-f) Other class-wise perturbations are very effective at corrupting network representations during training – so effective that even DFR is unable to recover test accuracy from random features (red dotted line).
A.3 Additional Results for Section 4.3: Linearly Separable Perturbations Are Not Necessary
A.3.1 Other Background
A related unlearnable dataset introduces entangled features (EntF) [34] which is motivated by the separability of recent poisoning perturbations. However, separability is qualitatively evaluated through t-SNE visualizations, which is different from the separability experiment we perform. More specifically, t-SNE cluster separability should not be equated to the linear separability we measure in Table 2 because it is possible to have linearly separable data that, when plotted using t-SNE, appears not separable. In other words, the EntF poison from [34] could still contain linearly separable perturbations.
A.3.2 Evaluating Linear Separability of Poison Images
In Section 4.3, we document the linear separability of perturbations from various poisons, as in [36]. Poison images, on the other hand, behave slightly differently. In Table 6, we report logistic regression train accuracy on various CIFAR-10 poison images. We find that Unlearnable Examples,  $LSP$ ,  $OPS + EM$ , and class-wise poisons have linearly separable poison images, but the remaining poisons we consider do not.
A.4 Additional Section 4.4 Results: Orthogonal Projection for Learning From Datasets with Class-wise, Linear Perturbations
A.4.1 ViT for Section 4.4