Efficient Availability Attacks against Supervised and Contrastive Learning Simultaneously

Authors: yihan  wang
yifan  zhu
xiao-shan  gao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 17:03:53

Abstract: Availability attacks provide a tool to prevent the unauthorized use of private data and commercial datasets by generating imperceptible noise and crafting unlearnable examples before release. 
Ideally, the obtained unlearnability can prevent algorithms from training usable models. 
When supervised learning (SL) algorithms have failed, a malicious data collector possibly resorts to contrastive learning (CL) algorithms to bypass the protection.
Through evaluation, we have found that most existing methods are unable to achieve both supervised and contrastive unlearnability, which poses risks to data protection by availability attacks.
Different from recent methods based on contrastive learning, we employ contrastive-like data augmentations in supervised learning frameworks to obtain attacks effective for both SL and CL.
Our proposed AUE and AAP attacks achieve state-of-the-art worst-case unlearnability across SL and CL algorithms with less computation consumption, showcasing prospects in real-world applications. 
The code is available at https://github.com/EhanW/AUE-AAP.

Main paper: 
 introduction: 
 Availability attacks [2] add imperceptible perturbations to training data, making the subsequently trained model unavailable. The motivations behind this kind of data poisoning attack involve protecting private data and commercial datasets from unauthorized use [22] . For example, according to a report [19] , a tech company illegally obtained over 3B facial images as the training set to develop a commercial facial recognition model. In this type of scenario, availability attacks provide tools to process user images before release, preserving legibility but impeding subsequent training. In particular, Huang et al. [22] reduces the accuracy of face recognition of 50 identities in WebFace [54] from 86% to 16%. In recent years, various availability attacks against supervised learning (SL) [11-13, 41, 56, 28] have been proposed. Meanwhile, contrastive learning (CL) allows people to extract meaningful features from unlabeled data in a self-supervised way. After subsequent linear probing or fine-tuning, CL algorithms have achieved comparable accuracy or even surpassed the performance of SL [5, 7, 15, 6] . However, most attacks designed for poisoning SL are ineffective against CL, as shown in Figure 1 . It sheds light on a potential issue of using availability attacks to protect data: a malicious data collector can traverse both supervised and contrastive algorithms to effectively leverage collected data. Hence, we introduce worst-case unlearnability (see Section 3.1) as the evaluation metric for availability attacks to emphasize the demand to deal with a trickier unauthorized data Figure 2 : Illustration of our proposed method. Separated by a vertical dashed line, the left side shows the process of generating the poisoning attack, while the right side depicts the training process on the poisoned dataset. On the generation side, above the horizontal dashed line are the existing methods based on contrastive error minimization, while below the dashed line are our proposed methods based on supervised error minimization/maximization (the blue flow). Our attack leverages the stronger contrastive augmentations to obtain effectiveness against both supervised learning and contrastive learning algorithms. Label information is involved in both our method and CL-based methods.collector. In recent years, contrastive error minimization attack is proposed to poison contrastive learning [16] , and then label-dependent components are incorporated into it to simultaneously achieve supervised unlearnability besides contrastive unlearnability [36, 29] . However, compared to SL-based ones, these CL-based attacks lack efficiency in poisoning generation, potentially hindering availability attacks from protecting extensive data in the real world (see Section 5.3) . Therefore, an effective as well as efficient availability attack against both SL and CL is imminent. Specifically, our motivation for this paper comes from two aspects: 1) A fully functional availability attack needs to be effective against subsequent supervised and contrastive learning algorithms simultaneously. 2) Attacks based on supervised learning can be superior in efficiency compared to those based on contrastive learning. To design a non-CL-based attack that possesses both supervised and contrastive unlearnability, we start from an interesting observation that supervised training with contrastive data augmentations mimics contrastive training to some extent (see Section 4.1). As shown in Figure 2 , this technique of enhancing data augmentations can be easily embodied in two basic supervised attack frameworks, i.e., error-minimization, and error-maximization, resulting in our proposed AUE and AAP attacks (see Sections 4.2 and 4.3). Enhanced augmentations allow us to craft perturbations for a contrastive-like reference model. These perturbations implicitly adapt to the contrastive training process and then learn deceptive patterns that fool contrastive learning. The supervised unlearnability is still preserved since the generation process is based on supervised optimization.On experimental side, we evaluate the standard supervised learning algorithm and four representative contrastive learning algorithms, SimCLR [5] , MoCo [7] , BYOL [15] and SimSiam [6] . Our proposed AUE and AAP attacks achieve the state-of-the-art worst-case unlearnability on CIFAR-10/100 and Tiny/Mini-ImageNet datasets (see Section 5.2). Specifically, our method exhibits excellent performance on the ImageNet-100 as presented in Figure 3 , showcasing its prospects in real-world applications. Meanwhile, unlike methods that add additional components to the contrastive errorminimization framework, we modify the data augmentation in the simpler supervised attack frameworks, following a minimalist approach to algorithm design. Benefiting from this, our methods are more efficient, while delivering better performance. We summarize our contributions as follows:• We evaluate existing availability attacks and point out the potential security risks of using them to protect data when facing data abusers who will traverse both supervised and contrastive learning algorithms.• We start from supervised poisoning approaches and enhance data augmentations to attain attacks against both supervised and contrastive learning.Let D ′ c be a poisoned version of a clean dataset D c . The alignment gap and uniformity gap between clean and poisoned datasets are defined as follows:AG = A(D c ) -A(D ′ c ), UG = U(D c ) -U(D ′ c ).(1) Intuitively, these gaps characterize the difference between clean features and poisoned features. We will check the relationship between these gaps and contrastive unlearnability in Section 3.2.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: We summarize our work in the abstract and list our contribution at the end of the introduction. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
basic availability attacks: 
 The essence of availability attacks is to prevent a trained model from well generalizing to clean data. We will revisit two representative approaches to poisoning supervised learning. Error minimization. Unlearnable example attack (UE) generates poisoning by alternately optimizing the reference model and perturbations [22] :EQUATIONwhere f is a classifier, L SL (•, •; •) is the supervised loss, D c is a dataset to be processed and δ is a poisoning map.Error maximization. Adversarial poisoning (AP) optimizes perturbations through a pre-trained classifier to equip them with non-robust but useful features from a different label [13] :EQUATIONs.t. f * ∈ arg min f E Dc L SL (x, y; f )] ,where K is the label shift. Self-ensemble protection (SEP) generates adversarial poisons using several checkpoints to improve attack performance [4] .Contrastive error minimization. Contrastive poisoning (CP) extends the error minimization framework to contrastive error minimization to poison contrastive learning [16] :EQUATIONwhere g is an encoder and L CL (•; •) denotes the contrastive loss for simplicity. Later, the transferable unlearnable example attack (TUE) introduces a regularization term called class-wise separability discriminant to equip CP noises with supervised unlearnability [36] . Then, transferable poisoning (TP) combines contrastive error minimization with supervised adversarial poisoning to obtain both supervised and contrastive unlearnability [29] . It is worth mentioning that both TUE and TP leverage label information in their proposed schemes, while CP requires no label information but lacks stable effect on supervised learning.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: In Section 5.2, we discuss limitations of the AAP attack. In Appendix D, we discuss challenges inherited from basic availability attacks such as defensive methods and poisoning ratio sensitivity. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
threat model: 
 In our threat model, an unauthorized data collector assembles labeled data into a dataset. The access to label information is reasonable since the collector can crawl individual images from certain accounts or steal (and annotate) a commercial dataset. A data publisher is supposed to process data before release using an availability attack such that processed data is resilient to subsequent supervised learning algorithms as well as contrastive learning algorithms adopted by the data collector. We will define worst-case unlearnability and discuss the contrastive unlearnability of existing attacks.Suppose an unprocessed dataset D c is i.i.d sampled from a data distribution D. An availability attack δ maps a data-label pair (x, y) ∈ D c to a noise δ(x, y) within an L p -norm ball B p (ϵ). In this paper, we set p = ∞ and ϵ = 8/255. It results in a protected dataset D ′ c = {(x + δ(x, y), y)|(x, y) ∈ D c } to which a data collector has only access. For potential algorithms, we refer f to a supervised learning classifier and g to a contrastive learning encoder beyond which is a linear probing head h. The goal of the data publisher is to find a poisoning map δ that significantly degrades the generalization performance of both f δ and h δ • g δ which are trained on D ′ c . We define the worst-case unlearnability across supervised and contrastive learning algorithms of the following form:EQUATIONs.t. f δ ∈ arg min f E Dc L SL (x + δ(x, y), y; f ) , g δ ∈ arg min g E Dc L CL (x + δ(x, y); g) , h δ ∈ arg min h E Dc L SL (x + δ(x, y), y; h • g δ ) .It is a fair metric that accurately depicts scenarios facing more cunning data abusers in the real world.In contrast, other metrics, such as average-case unlearnability, can be heavily influenced by the attack's strong preference for a certain algorithm. Our threat model differs from the setting adopted by He et al. [16] in which the linear probing stage relies on the unprocessed clean data as downstream tasks; see more discussion in Appendix D.10. DC [11] 0.12 0.07 86.1 UE [22] 0.05 0.03 89.0 AR [41] 0.07 0.09 88.8 NTGA [58] 0.12 0.12 86.9 SN [56] 0.08 0.00 90.6 OPS [52] 0.04 0.01 86.7 GUE [28] 0.07 0.03 88.8 REM [14] 0.12 0.04 88.6 EntF [51] 0.01 -0.04 87.5 HYPO [47] 0.11 0.13 86.9 AP [13] 0.18 0.44 48.4 SEP [4] 0.24 0.25 37.3 CP [16] 0.55 0.87 38.7 TUE [36] 0.30 0.76 48.1 TP [29] 0.52 0.82 31.4In Table 1 , we evaluate the attack performance of existing poisoning approaches against the SimCLR algorithm on CIFAR-10 and ResNet-18. To better understand contrastive unlearnability, we also check alignment and uniformity gaps between clean and poisoned data defined in Equation (1) Prominent gaps indicate a significant difference between clean and poisoned feature distributions. Thus, no matter how well the classifier performs on poisoned features, it hardly generalizes to clean features and the attack is successful. In contrast, small gaps likely imply that clean features are similar to poisoned features. When gaps are small, the test accuracy is high and the attack fails.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We use public datasets in experiments and provide code in the supplementary material.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
method: 
 Since contrastive loss is related to alignment and uniformity [50] , the contrastive error minimization (CP) attack optimizes the loss directly and obtains contrastive unlearnability. Beyond this, TUE and TP incorporate additional label-dependent components to obtain supervised unlearnability. However, optimizing contrastive loss is very time and memory-consuming, impeding their applications in realworld scenarios. Different from them, we start from more efficient supervised poisoning frameworks instead to achieve both supervised unlearnability and contrastive unlearnability simultaneously.The key point to get there is data augmentation. In the rest of this section, we first illustrate how contrastive learning data augmentations help mimic contrastive learning with supervised models through empirical observations and intuition from a toy example. In other words, enhancing data augmentation helps supervised learning implicitly optimize the contrastive loss. Then we combine this very effective technique with supervised error minimization and maximization frameworks and propose augmented unlearnable examples (AUE) attacks and augmented adversarial poisoning (AAP) attacks. In other words, supervised error minimization mimics contrastive error minimization to some extent. Therefore, incorporating stronger data augmentation potentially enables availability attacks based on supervised error minimization or maximization to deceive contrastive learning.To provide more intuition about this idea, we give a toy example and have a closer look at the relationship between supervised loss and contrastive loss. For a supervised model f = h • g, assume g is a normalized feature extractor, h is a square full-rank linear classifier, D is a balanced distribution, L SL is MSE loss, training error E SL = E L SL , and L CL contains only one negative example. In this toy example, if LCL and LSL employ the same data augmentation and f is well-trained, it holds with high probability that L CL < l(E SL ), where l(•) is an increasing function. In other words, the upper bound of contrastive loss decreases as the supervised loss decreases. We have a more detailed and formal discussion on this toy example in Appendix E.Based on these interesting observations, instead of adding components to contrastive error minimization to achieve supervised unlearnability, we opt for deriving stronger contrastive unlearnability from supervised error minimization and maximization.Recall unlearnable examples (UE) are generated by supervised error minimization which alternately updates a reference model and noises in Equation ( 2). Now we employ contrastive-like strong data augmentation distribution µ and add perturbations in a differentiable way, i.e., π(x + δ(x, y)), π ∼ µ. As discussed in the previous section, minimizing the augmented supervised loss L SL (π(x + δ(x, y), y; f ) implicitly minimizes the contrastive loss L CL (x + δ(x, y); g) which appears in contrastive error minimization, i.e., Equation (4). In other words, supervised error-minimizing noises with enhanced data augmentation can partially replace the functionality of contrastive error-minimizing noises to deceive contrastive learning. δi ← 0, i = 1, 2, • • • , r ▷ Initialize perturbations for t = 1, • • • , T do for t θ = 1, • • • , T θ do▷ Update the reference model Sample a data batch {(x l j , y l j )} m j=1 and an augmentation batch Require: Similar to the setting in Algorithm 1.{π l j ∼ µs} m j=1 θ ← θ -α θ m • m j=1 ∇ θ LSL(π l j (x l j + δ l j ), y l j ; f θ ) for t δ = 1, • • • , T δ do ▷ Update perturbations Sample a data batch {(x l j , y l j )} m j=1 for tp = 1, • • • , Tp do Sample an augmentation batch {π l j ∼ µs} m j=1 δ l j ← Clip ϵ δ l j -α δ • sign(∇ δ l j LSL(π l j (x l j + δ l j ), y l j ; f θ )) , j = 1, 2, • • • , m 4.3 Augmented Adversarial Poisoning (AAP)Ensure:Perturbations {δi} r i=1 δi ← 0, i = 1, 2, • • • , r ▷ Initialize perturbations for t = 1, • • • , T do ▷ Update the reference model for t θ = 1, • • • , T θ doSample a data batch {(x l j , y l j )} m j=1 and an augmentation batch{π l j ∼ µs} m j=1 θ ← θ -α θ m • m j=1 ∇ θ LSL(π l j (x l j ), y l j ; f θ ) for i = 1, • • • , r do ▷ Update adversarial examples for tp = 1, • • • , Tp do Sample πi ∼ µs δi ← Clip ϵ δi -α δ • sign(∇ δ i LSL(πi(xi + δi), yi + 1; f θ ))5 ExperimentsWe will evaluate the worst-case unlearnability of our proposed AUE and AAP attacks on multiple datasets and compare the poisoning generation efficiency with other methods. Besides, we will check the efficacy of our method against more evaluation algorithms and the transferability across network architectures. Then we will perform an ablation study of decoupling argumentation components in our method. 
setup: 
 We conduct experiments on CIFAR-10/100 [26] , Tiny-ImageNet [27] , modified Mini-ImageNet [49] , and ImageNet-100 [38] . ResNet-18 [17] is used for poison generation and evaluation if not otherwise stated. Our threat model considers the worst-case unlearnability across supervised and contrastive (self-supervised) algorithms including standard SL, SimCLR, MoCo, BYOL, and SimSiam. We implement linear probing on the encoder to evaluate contrastive unlearnability.We adopt AP, SEP-FA-VR, CP, TUE, and TP as baseline methods for the worst-case unlearnability.Since the generation of untargeted adversarial poisoning is unstable [13] , AP and AAP attacks are targeted if not otherwise stated (see more discussion in Appendix D.5). In particular, only CIFAR-10 results in Table 3 report untargeted AP and AAP. For CP and TUE attacks, we report the best results across the CL algorithms they depend on (see additional results in Appendix D.6). Detailed settings for attack implementation and evaluation are shown in Appendix C. Worst-case unlearnability. In Table 3 , our AAP attack achieves the best worst-case unlearnability on CIFAR-10 and both AAP and AUE attacks outperform other methods on CIFAR-100. Particularly, AAP improves the performance by 8.5%/7.5% on CIFAR-10/100 and AUE becomes better than AAP on CIFAR-100. In other methods, CP loses supervised unlearnability on CIFAR-100 and TUE is better than TP on both datasets. Furthermore, we evaluate attacks on higher-resolution datasets including Tiny-ImageNet (64x64) and Mini-ImageNet (84x84) in Table 4 . On both datasets, AUE outperforms other methods in terms of the worst-case unlearnability. Besides, AUE also achieves the best supervised unlearnability. For AAP, its worst-case unlearnability is better than AP but not as good as TUE. Moreover, compared to AP, AAP suffers a trade-off between supervised unlearnability and contrastive unlearnability.Comparison between AUE and AAP. On simpler datasets (low resolution, few classes), AAP has an advantage over AUE. However, on more complex datasets (high resolution, many classes), AUE outperforms AAP. One possible reason for this could be that optimizing AAP is more challenging. Firstly, generating adversarial poisoning inherently depends on a well-performing reference classifier.For instance, Fowl et al. 13 uses a pre-trained ImageNet model, whereas in this paper we train classifiers from scratch. Additionally, stronger data augmentation used in the reference model training can affect its accuracy, which in turn impacts the quality of the generated adversarial perturbations.Enhancing the performance of AAP is an interesting direction for future work.Algorithm transferability. CL-based methods face the issue of transferability from the generation CL algorithm and the evaluation CL algorithm. For example, TP is generated using the SimCLR, which performs well against SimCLR evaluation, but its performance sharply declines when tested with BYOL. The same phenomenon also occurs with TUE and CP and their worst-case unlearnability is highly dependent on the appropriate generation algorithm, which you can check in Appendix D. 6 .In contrast, our SL-based attacks get rid of this issue because their poisoning generation involves no CL algorithms. In real-world scenarios, availability attacks need to generate perturbations for accumulating data as quickly as possible. For expanding datasets, like continually updated social media user data, the poisoning used for data protection also needs to be updated periodically. Since contrastive learning involves larger batches (e.g., 512) and a longer training process (e.g., 1000 epochs), these contrastive error minimization-based attacks require more time and memory consumption to generate perturbations.In Figure 6 , we report the time cost of poisoning CIFAR-10/100 using the same device. Baseline methods adopt their default configurations. Our supervised learning-based approaches are 3x, 6x, and 17x faster than TUE, CP, and TP. Additionally, our methods admit smaller batches and simpler cross-entropy loss which require less memory, allowing for the generation of availability attacks on larger datasets with fewer devices. Refer to Appendix D.1 for more results about the efficiency of our methods. Overall, our method is more promising than CL-based methods due to the time and memory efficiency in real-world applications. Besides linear probing, we also apply the k-nearest neighbors (k-NN) algorithm to the feature space to evaluate the contrastive unlearnability. In Table 5 , both AUE and AAP prominently reduce the k-NN accuracy. It indicates that the features of poisoned training inputs largely differ from those of clean test inputs. Non-robust features in imperceptible perturbations heavily affect the encoder's behavior and hinder its generalization ability.In addition to supervised learning and contrastive learning algorithms, we consider two more CL-like algorithms including supervised contrastive learning, i.e., SupCL [24] and a semi-supervised learning algorithm FixMatch [45] . FixMatch uses WideResNet [59] and detailed settings are in Appendix C.5. Table 5 demonstrates that our attacks are still effective against SupCL and FixMatch. It indicates that our methods can handle more variants derived from supervised learning and contrastive learning algorithms.Since the data protector is unaware of networks used in future training, availability attacks should be effective for different architectures. We generate AUE and AAP using ResNet-18 and test them on ResNet-50, VGG-19 [44] , DenseNet-121 [21] , MobileNet v2 [20, 40] , and ViT [10] . In Table 6 , both supervised and contrastive unlearnability of AUE and AAP can transfer across these architectures.In settings of AUE and AAP, we control the strength of ResizedCrop, ColorJitter, and Grayscale through a single strength hyperparameter s for the poison generation, as shown in Code 1. In Table 7 , we decouple the strength hyperparameters for these three random transforms and evaluate the resulting attacks against SimCLR. Different factors show different influences on the contrastive unlearnability for AUE and AAP. For example, enhancing ResizedCrop strength alone is less effective than enhancing Grayscale alone in AUE generation. However, adjusting three factors together generally outperforms other options in conclusion.Table 7 : SimCLR accuracy(%) of attacks generated with decoupled strength parameters on CIFAR-10. For example, 0-0-s means that ResizedCrop strength is 0, ColorJitter strength is 0, and Grayscale strength is s.Attacks 0-0-0 0-0-s 0-s-0 s-0-0 0-s-s s-0-s s-s-0 s-s-s When generating availability attacks, the gradient of perturbations is often computed through data augmentations. In literature, SL-based attacks generally use mild supervised data augmentation, i.e., RandomCrop and RandomHorizontalFlip [13] . The expectation over transformation (EOT) technique [1] adopted by Fu et al. [14] first samples several such mild augmentations and then computes the average gradient over them. Note that our proposed method is not a variant of EOT. CL-based attacks use contrastive augmentations [16, 36, 29] . To our knowledge, we are the first to use contrastive-like strong data augmentations in SL-based poisoning frameworks. Besides, we provide additional related works on availability attacks in Appendix A.AUE 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: These setting and details are described in Appendix D.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 Since contrastive learning algorithms bring new challenges to protect data using availability attacks, we explore effective attacks against both supervised and contrastive learning. We introduce a very effective modification of data augmentation in supervised poisoning frameworks and propose attacks achieving superior performance and efficiency compared to existing methods, offering more potential in real-world applications. Considering availability attacks still face obstacles such as adversarial training mitigation and poisoning ratio sensitivity, addressing these challenges while maintaining both supervised and contrastive unlearnability will be an important direction for our future research.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: In Tables 3 and 4 , we report accuracy after 3 evaluations with random seeds for fair comparison. Due to space constraints, we did not report error bars in the table, which is common in papers within this field.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We report computation consumption of our method in Appendix D.1.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss in the paper that the availability attacks provide a tool to protect private and commercial data.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We use public GitHub repositories and public packages. We cite them in our paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: Our new assets are well documented.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Availability attacks for supervised learning include error-minimizing noises [22] , adversarial example poisoning [13, 4] , neural tangent generalization attack [58] , generative poisoning attack [11] , autoregression perturbation [41] , one-pixel perturbation [52] , convolution-based attack [39] , synthetic perturbation [56] , and game-theoretic unlearnable examples [28] . Yu et al. [56] illustrated linearly separable perturbations work as shortcuts for supervised learning. Robust error-minimizing noises [14] , entangled features strategy [51] , and hypocritical perturbation [47] were designed to deceive adversarial training. Contrastive poisoning [16] aimed at poisoning contrastive learning. Transferable unlearnable examples [36] and transferable poisoning [29] improved the supervised unlearnability of contrastive poisoning. Zhang et al. [60] proposed to generate label-agnostic noises with cluster-wise perturbations. Chen et al. [3] introduced CLIP-guided unlearnable perturbation generators that can transfer across different datasets.On the defense side, adversarial training can largely mitigate the unlearnability [46] . Liu et al. [30] , Qin et al. [35] , Zhu et al. [61] leverages crafted data augmentations as defense. Sandoval-Segura et al. [42] suggests that the orthogonal projection technique is effective against class-wise attacks.Diffusion models have been proposed to purify unlearnable perturbations [23, 9] . Yu et al. [57] proposed a VAE-based purification method that requires no additional clean data. Qin et al. [34] introduced a benchmark for availability attacks.Here we introduce Info-NCE-based contrastive learning. As shown in Code 1, strong augmentation is a key component in contrastive learning. Different augmented views of an image focus on various details. If x, x pos are two augmented views of the same image, we say (x, x pos ) is a positive pair. Conversely, a negative pair (x, x neg ) contains two augmented views of two different images. Given an encoder g, we denote z = g(x) for simplicity and the InfoNCE loss is defined as:L InfoNCE = 1 N i s(z i , z ′ i ) 1 N j s(z i , z ′ j ), where {(z i , z ′ i )} N i=1 is a set of features of positive pairs and {(z i , z ′ j )} N j=1 is a set of features of negative pairs for each x i ; the function s(z, z ′ ) = exp( z•z ′⊤ T ||z||||z ′ || ) with a temperature parameter T . This object function aims to maximize the cosine similarity of positive pairs while minimizing the cosine similarity of negative pairs. ImageNet-100. ImageNet-100 is a subset of ImageNet-1k Dataset from ImageNet Large Scale Visual Recognition Challenge 2012 [38] . It contains 100 random classes. The training set has 130,000 images. The test set has 5,000 images. Images are processed to 224x224 colored ones as input data to models.ResNet. On CIFAR-10/CIFAR-100, we set the kernel size of the first convolutional layer to 3 and removed the following max-pooling layer. On other datasets, we do not modify the models.In Code 1, we show the different implementations of data augmentation between supervised learning and contrastive learning. For supervised learning, we consider the typical augmentations including Crop and HorizontalFlip. For contrastive learning, we consider the typical augmentations including ResizedCrop, HorizontalFlip, ColorJitter, and Grayscale, and its default strength s = 1. In the generation process of our AUE and AAP attacks, we replace the supervised augmentations with contrastive-like augmentations of a strength parameter s.Code Listing 1: Different data augmentations used in supervised learning and contrastive learning on CIFAR-10/100 datasets. The intensity of contrastive augmentations can be adjusted via strength s. We leverage differentiable augmentation modules in Konia2 [37] which is a differentiable computer vision library for PyTorch. The contrastive augmentations for Tiny/Mini-ImageNet and ImageNet-100 are similar to those for CIFAR-10/100 in Code 1 but only adapt the image size. We train the reference model for T = 40 epochs, and the initial learning rate α θ is 0.5. The PGD process for noise generation takes T p = 250 steps with step size α δ = 0.08/255. Other settings are the same as AUE. The label translation is K = 1. The augmentation strength s = 0.4 for CIFAR-10 and s = 0.8 for CIFAR-100, Tiny-ImageNet, Mini-ImageNet, and ImageNet-100.Besides targeted AP and AAP attacks described in Equation (3) and Algorithm 2, untargeted attacks refer to maximizing the loss between the image and its true label, rather than minimizing the loss between the image and a shifted label. We only report untargeted attack results in Table 3 for CIFAR-10 and discuss them more in Appendix D.5.Sample-wise Attack. When a poisoning map δ(x, y) only depends on label y, the resulting attack is called a class-wise attack; otherwise, it is a sample-wise attack. In this paper, we generate sample-wise attacks.A CP attack implements contrastive error minimization on a specific contrastive learning algorithm. There are three specified attacks including CP-SimCLR, CP-MoCo, and CP-BYOL. Similarly, TUE has three specified attacks including TUE-SimCLR, TUE-MoCo, and TUE-SimSiam. We report the best results of CP attacks and TUE attacks according to the worst-case unlearnability. Besides, we provide detailed results in Appendix D.6 for more discussion.Contrastive learning. The setup for SimCLR, MoCo, BYOL, and SimSiam are shown in Table 8 . The 100-epoch linear probing stage uses an SGD optimizer and a scheduler that decays 0.2 at 60, 75, and 90 epochs. The probing learning rate is 1.0 for SimCLR, MoCo, BYOL, and 5.0 for SimSiam on CIFAR-10/100, Tiny/Mini-ImageNet. On ImageNet-100, the unsupervised contrastive learning optimizes 200 epochs and the linear probing uses a learning rate of 10.0. Other settings are the same as other datasets. After generating our attacks on CIFAR-10/100, we report average test accuracy after 3 evaluations with random seeds. Supervised learning. We augment the training data by RandomHorizontalFlip and RandomCrop with padding size l/8 on CIFAR-10/100 and Tiny/Mini-ImageNet. l is the image size. On ImageNet-100, we augment using RandomResizedCrop and RandomHorizontalFlip.SupCL and FixMatch. We use ResNet-18 for SupCL evaluation on CIFAR-10 and CIFAR-100. For FixMatch evaluation, we use WideResNet-28-2 and 4000 labeled data on CIFAR-10; we use WideResNet-28-8 and 10000 labeled data on CIFAR-100. We report the time consumption of generating AUE and AAP attacks. For CIFAR-10/100, Tiny/Mini-ImageNet, experiments are conducted using a single NVIDIA GeForce RTX 3090 GPU. For ImageNet-100, experiments are conducted using a single NVIDIA A800 GPU. On CIFAR-10/100, AUE/AAP costs around 2.7/2.2 hours. On Mini-ImageNet, AUE/AAP costs around 2.5/2 hours. On Tiny-ImageNet, AUE/AAP costs around 2.5/3.8 hours. On ImageNet-100, AUE/AAP costs around 12/10 hours. In comparison, on CIFAR-10/100 and using the same device, CP-SimCLR costs around 48 hours, TUE-MoCo costs around 8.5 hours, and TP costs around 16 hours to generate poisons. Our supervised poisoning attacks are much more efficient than contrastive poisoning attacks. We scale imperceptible perturbations from [-8/255, 8/255] to [0,1] and show their images in Figure 8 . Enhanced data augmentations endow AUE with more complicated patterns than UE. In terms of frequency, they are more high-frequency than UE. Since contrastive augmentations include grayscale that squeezes low-frequency shortcuts [30] , attacks against CL first need to come through them and thus prefer high-frequency patterns. Moreover, we check the class-wise separability of perturbations using t-SNE visualization [48] in Figure 8 . Perturbations from AUE and T-AAP are less separable than those from UE and T-AP and coincide with the characteristics of perturbations from contrastive error minimization [16] . AAP. We gradually increase the data augmentation strength s in the supervised error maximization according to Algorithm 2. In Figure 9b , the SimCLR accuracy decreases with the strength, while the supervised learning accuracy slightly increases. Proper augmentation strengths improve the contrastive unlearnability but too large s might introduce difficulty in poison generation and harm the supervised unlearnability. We select s = 0.4 for CIFAR-10 and s = 0.8 for CIFAR-100.Strength and Gaps On CIFAR-10, we gradually increase the augmentation strength from 0 to the default setting, i.e. s = 0.6 in the generation of AUE attacks and evaluate the alignment gaps, uniformity gaps, and the SimCLR Accuracy in Table 9 . In this case, the larger the gaps, the lower the accuracy of SimCLR.Instead of Equation ( 3), the untargeted attack refers to the following optimization:max δ E Dc L SL (x + δ(x, y), y; f * ) .In a targeted attack, the perturbations for a class of data are optimized to fit another label, so they finally contain the non-robust feature of the target class. However, there is no consistent target label for a class of data in an untargeted attack. Since availability attacks create shortcuts for the classification task [56] , the untargeted attack becomes more difficult than the targeted one when the number of classes increases. [13] also reports that the generation of untargeted attacks is unstable and they focus on targeted attacks on more complex datasets. Thus, we only perform untargeted AAP attacks on the simple dataset CIFAR-10 and report the results in Table 3 . As a complement to it, we present the performance of targeted attacks on CIFAR-10 in Table 10 , where the untargeted attacks are better than targeted ones in the worst-case unlearnability. In the main body, we consider the poisoning attacks constrained in a L ∞ -norm ball with radius 8/255. The constraint is to ensure perturbations are imperceptible to human eyes. We investigate the influence of different poisoning budgets. AUE and AAP attacks are generated with poisoning budgets of 2/255, 4/255, 6/255 and are evaluated by SL and SimCLR. In Table 12 , the larger the poisoning budgets, the better the attack performance. Availability attacks are sensitive to the proportion of poisoned data in the dataset and usually need to poison the whole dataset [22, 13] . In the main body, we report results when the poisoning ratio is 100%. Here, we investigate the influence of the poisoning ratio on the attack performance of AUE and AAP. Table 13 illustrates that our augmented methods inherit the vulnerability to poisoning ratio from basic approaches, i.e. UE and AP, though AUE is more robust than AAP. This characteristic also necessitates the prompt processing of newly acquired clean data, imposing higher efficiency demands on the generation of attacks.On the defense side against availability attacks, AT [31] and AdvCL [25] ) applied adversarial training in supervised learning and contrastive learning respectively; ISS [30] and UEraser [35] leveraged designed data augmentations to eliminate supervised unlearnability; AVATAR [9] employed a diffusion model to purify poisoned data. In Table 14 , we evaluate our attacks through these defense methods as well as SimCLR with Cutout [8] , Random noise, and Gaussian Blur. The defensive budget for AT and AdvCL is 8/255; the length parameter for Couout is 8; the kernel size for Gaussian Blur is 3; the variance for Random noise is 8/255.The defense performance of a method differs when facing different attacks. For example, UEraser can recover the accuracy of TUE-SimCLR from 10.6% to above 92.7%, while its effect on our AUE attack is much weaker. At the cost of a significant amount of extra training time, adversarial training, i.e. AT and AdvCL, can increase accuracy to around 80%. ISS mitigates the supervised unlearnability of evaluated attacks back to levels close to 85%, but its Grayscale component may even have negative effects. Gaussian Blur is more effective than Cutout and Random noise for contrastive learning.Recently, diffusion models have provided a powerful tool to purify image perturbations [55, 32, 43 ].Here we evaluate AVATAR which employs a diffusion model trained on the CIFAR-10 training dataset. From the table, AVATAR generally achieves the best defense against our proposed attacks, but the final accuracy still exhibits a gap compared to training with clean data. We believe it's an interesting and worthy future direction to improve attacks' resilience to potential defenses while maintaining algorithm transferability While our threat model linear probes on poisoned data, He et al. [16] use clean data for linear probing instead. In Figure 10 , we compare the final classification performance of SimCLR models in these two settings. Feature extractors are trained on poisoned data and are fixed. We focus on the classification performance after linear probing on clean or poisoned data. While CP and TUE obtain similar attack performance in both cases, clean linear probing can mitigate SL-based attacks including AP, SEP-FA-VR, AAP, and AUE. On one hand, for SL-based poisoning, the dissimilarities between clean features and poisoned features hinder a classifier head obtained by poisoned linear probing in generalizing to clean features, as discussed in Section 3.2. However, clean features still contain useful information and can derive another classifier head to perform classification. On the other hand, contrastive error-minimizing noises confuse the feature extractor directly such that even clean data fail to activate useful features for classification. But in general, given a responsible data publisher who protects data using availability attacks before release, an unauthorized data collector has no access to unprocessed data for clean linear probing. Thus, it is sufficient to achieve contrastive unlearnability with poisoned linear probing in real scenarios.We study a model f = h • g : R d → R n with a normalized feature extractor g : R d → R n such that ||g(x)|| ≡ 1 and a full rank linear classifier h : R n → R n in the sense that h(z) = W z + b with a full rank square matrix W ∈ R n×n . By singular values decomposition (SVD),W = U ΣV with orthogonal matrices U, V ∈ R n×n and Σ = diag(σ 1 , • • • , σ n ), σ 1 ≥ • • • ≥ σ n > 0.Let D be a balanced data distribution, i.e. each class would be sampled with the same probability, D x be the margin distribution, and µ be an augmentation distribution. Assume the supervised loss L SL is the mean squared error, and the contrastive loss L CL contains only one negative example:L SL (x, y, π) = 1 n ||h • g(π(x)) -e y || 2 L CL (x, x -, π, τ, ρ) = log(1 + e g(π(x)) ⊤ g(ρ(x)) e g(π(x)) ⊤ g(τ (x -)) ). Proposition E.1. Let E SL = ED,µ L SL (x, y, π) . With probability at least 1 -4 √ E SL , it holds LCL(x, x -, π, τ, ρ) < 1 n log(1 + σn σn -2n √ ESL ) + n -1 n log(1 + σ 2 1 σn -σn(1 -2n √ ESL) 2 σ 2 1 σn -2nσ 2 1 √ ESL ).Remark E.2. 1) Assumptions of a square matrix and positive singular values are necessary. Otherwise, the dimensional reduction of feature space impairs the relation between supervised and contrastive losses. 2) Since supervised losses contain limited information about negative pairs, this inequality is naturally loose. However, in the case that supervised learning fits very well, it at least implies that positive features g(τ (x)) are closer to g(π(x)) than negative features g(ρ(x -)).Lemma E.3. For any z ∈ R n ,σ n ||z|| ≤ ||W z|| ≤ σ 1 ||z||. Proof. Denote z = (z 1 , • • • , zn ) ⊤ = V z.Since orthogonal matrices preserve the norm,||W z|| = ||U ΣV z|| = ||Σz|| = n i=1 σ 2 i zi 2 , σ n ||z|| = σ n ||z|| ≤ n i=1 σ 2 i zi 2 ≤ σ 1 ||z|| = σ 1 ||z||. Lemma E.4. If E SL ≤ ϵ, then with probability at least 1 - √ ϵ ||h • g(π(x)) -e y || < n √ ϵ,where (x, y) ∼ D, π ∼ µ.AsE SL = E (x,y)∼D π∼µ 1 n ||h • g(π(x)) -e y || 2 , by Markov's inequality, it has Pr( 1 n ||h • g(π(x)) -e y || 2 ≥ √ ϵ) ≤ √ ϵ. Lemma E.5. If E SL ≤ ϵ, then with probability at least 1 -2 √ ϵ g(π(x)) ⊤ g(τ (x)) > 1 - 2n √ ϵ σ n , where x ∼ D x , π, τ ∼ µ.Proof. By Lemma E.4, with probability at least 1 -2 √ ϵ,||h • g(π(x)) -e y || < n √ ϵ and ||h • g(τ (x)) -e y || < n √ ϵ.By the triangle inequality,||h • g(π(x)) -h • g(τ (x))|| < 2 n √ ϵSince g is normalized, by Lemma E.3 we haveg(π(x)) ⊤ g(τ (x)) = 1 - 1 2 ||g(π(x)) -g(τ (x))|| 2 ≥ 1 - 1 2σ 2 n ||h • g(π(x)) -h • g(τ (x)|| 2 > 1 - 2n √ ϵ σ n .Lemma E.6. Assume D is a balanced dataset. If E SL ≤ ϵ, then with probability at least 1 -2 √ ϵ, one of the following two conditions holds 1. with probability n-1 n ,g(π(x)) ⊤ g(τ (x -)) < 1 - (1 -2n √ ϵ) 2 σ 2 1 ; 2. with probability 1 n , g(π(x)) ⊤ g(τ (x -)) ≤ 1.Proof.1. With probability n-1 n , for (x, y), (x -, y -) ∼ D, y ̸ = y -. By Lemma E.4, with probability at least 1 -2 √ ϵ,||h • g(π(x)) -e y || < n √ ϵ and ||h • g(τ (x -)) -e y -|| < n √ ϵ.By the triangle inequality,||g(π(x)) -g(τ (x -))|| ≥ 1 σ 1 ||h • g(π(x)) -h • g(τ (x -))|| ≥ 1 σ 1 (||e y -e y -|| -||h • g(π(x)) -e y || -||h • g(τ (x -)) -e y -||) > √ 2 -2 n √ ϵ σ 1 .Since g is normalized,g(π(x)) ⊤ g(τ (x -)) = 1 - 1 2 ||g(π(x)) -g(τ (x -))|| 2 < 1 - (1 -2n √ ϵ) 2 σ 2 1 .2. As we assume D is a balanced dataset, with probability 1 n , for (x, y), (x -, y -) ∼ D, y = y -. Since g is normalized,g(π(x)) ⊤ g(τ (x -)) = 1 - 1 2 ||g(π(x)) -g(τ (x -))|| 2 ≤ 1 - 1 2σ 2 1 ||h • g(π(x)) -h • g(τ (x -))|| 2 ≤ 1.E. L CL (x, x -, π, τ, ρ) = -log e g(π(x)) ⊤ g(τ (x)) e g(π(x)) ⊤ g(τ (x)) + e g(π(x)) ⊤ g(ρ(x -)) = log(1 + e g(π(x)) ⊤ g(ρ(x -)) e g(π(x)) ⊤ g(τ (x)) ) < n -1 n log(1 + 1 - (1-2n √ E SL ) 2 σ 2 1 1 - 2n √ E SL σn ) + 1 n log(1 + 1 1 - 2n √ E SL σn ) = 1 n log(1 + σ n σ n -2n √ E SL ) + n -1 n log(1 + σ 2 1 σ n -σ n (1 -2n √ E SL ) 2 σ 2 1 σ n -2nσ 2 1 √ E SL ).NeurIPS Paper ChecklistJustification: We read the code of ethics and obey it.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).https://github.com/kornia/kornia 


Decision: The paper introduces two novel attack methods, AUE (Augmented Unlearnable Examples) and AAP (Augmented Adversarial Poisoning), designed to thwart both supervised learning (SL) and contrastive learning (CL) algorithms. It paper addresses a crucial gap in the field: while availability attacks have been proposed to prevent unauthorized use of private data in SL settings, they many fail to be effective against CL algorithms. The paper received a generally positive response from the reviewers, who appreciated its timely and relevant contributions. However, concerns were raised regarding the novelty and depth of the technical contributions, as well as the robustness of the proposed methods under certain conditions. The authors provided a comprehensive rebuttal addressing most of the concerns raised by the reviewers. They clarified the differences between their work and prior approaches, particularly emphasizing the distinctiveness of their focus on cross-algorithm transferability rather than cross-dataset transferability. These efforts were generally well-received by the reviewers, leading to score adjustments in favor of the paper. Overall, this paper makes a meaningful contribution to the field of availability attacks by proposing methods that are both effective and efficient across supervised and contrastive learning paradigms. It would benefit from a more modest presentation of its novelty and an expanded discussion on the theoretical implications and potential limitations of the proposed methods.

Reviews: review: paper_summary: paper_summary: This paper aims to address the issue of effectively conducting availability attacks on both supervised learning (SL) and contrastive learning (CL) algorithms. Specifically, the paper highlights that existing methods fail to simultaneously achieve "unlearnability" for both SL and CL, posing risks to data protection. To tackle this challenge, the paper proposes a novel approach that employs contrastive-like data augmentation within the supervised learning framework to execute effective attacks on both SL and CL.
strength_weakness: strength_weakness: 1. The proposed AUE and AAP are all brittle under diffusion-based purification, which limits the practical usage of unlearnable examples in the real world. 

2. Lack of discussion and comparison with recent work [1] that leverages CLIP latent as guidance for crafting transferable and label-agnostic perturbation, which could potentially achieve efficient perturbation crafting for both SL and CL tasks. 

3. The results in Table 3 show that the proposed methods did not consistently outperform other methods under different SL and CL settings, which makes the stability of improvement questionable. 

4. This work essentially works by crafting perturbation that resilience to augmentations for both supervised learning and contrastive learning, which is a conceptually simple extension of the transformation augmentation technique in REM [2]. One of the fundamental limitations of such a direction is that unauthorized trainer might leverage stronger transformations like super-resolution or diffusion-based augmentation [3] in their training pipeline. 

Typos: Line 20: Change "particular, Huang et al. [20] reduces" to "particular, Huang et al. [20] reduced."

Ref.

[1]. One for All: A Universal Generator for Concept Unlearnability via Multi-Modal Alignment, ICML'24

[2]. Robust Unlearnable Examples: Protecting Data Against Adversarial Learning, ICLR'22'

[3]. DiffAug: Enhance Unsupervised Contrastive Learning with Domain-Knowledge-Free Diffusion-based Data Augmentation, ICML'24
questions: questions: Nowadays, one of the most popular ways to build classifiers is to leverage CLIP as a feature encoder with or without additional head to do classification. Is the perturbation crafted from AUE and AAP transferable to this setting?
limitations: limitations: Yes
paper_summary: paper_summary: This paper claims to introduce a threat scenario where the adversary may use contrastive learning to bypass the unlearnable examples that are crafted only for supervised learning. In this threat model, the authors showcase that previous works on unlearnable examples crafted only for supervised learning may fall short of contrastive learning. Based on this observation, the authors propose a new approach that considers both supervised and contrastive learning when generating unlearnable examples. The proposed approach is also efficient due to the augmentation adopted augmentation strategies. Extensive experiments show that the proposed method is effective in this new threat scenario.
strength_weakness: strength_weakness: The introduced topic is interesting but also limited. The authors point out that there is a lack of unlearnable approaches that are effective against both supervised learning (SL) and contrastive learning (CL). This observation is not fully novel, since CP, TP, and TUE have similar observations and are effective in most cases against both types of learning. For example, CP is an approach that is designed for contrastive learning but is also effective against supervised learning. I would suggest that the author could tone down the claims that hint at this observation as a novel finding. 

The main contribution of this work is to employ both SL and CL losses in the generation of unlearnable examples, including augmentations that imitate the CL in the SL scenario. The proposed approaches show better average and worst-case performance, but the improvement is incremental, according to Table 3 and Table 4. I would suggest that the authors could provide a deeper and more detailed analysis of the difference between SL and CL unlearnable examples by conducting more ablation studies on the augmentation strategies.  The working mechanism behind the augmentation strategies is of interest to the research community.
questions: questions: Please motivate the threat scenario where the proposed unlearnable examples are necessary. Further analysis of the augmentation strategies could be clarified.
limitations: limitations: The authors have discussed most limitations adequately.
paper_summary: paper_summary: This paper explores efficient availability attacks that target both supervised and contrastive learning algorithms simultaneously. The authors propose two attack methods, named AUE (Augmented Unlearnable Examples) and AAP (Augmented Adversarial Poisoning), which utilize enhanced data augmentations to craft effective poisoning attacks. These methods achieve state-of-the-art unlearnability with reduced computational demands, showing promise for practical applications. The paper evaluates the effectiveness of these attacks across multiple datasets and demonstrates their superiority over existing methods in achieving worst-case unlearnability while handling high-resolution datasets efficiently.
strength_weakness: strength_weakness: - While the motivation is justified, the technical contributions could be considered direct. Integrating data augmentation into the generation of poisons is a straightforward approach, which may not reflect a non-trivial technical contribution.
- The methods lack theoretical analysis, which raises questions about the guaranteed effectiveness of data augmentation in all scenarios. Additionally, there is no explanation of why data augmentation works effectively or if it completely addresses the issue at hand through augmentation alone.
questions: questions: What is the biggest challenge in this work?
limitations: limitations: None
paper_summary: paper_summary: This paper studies the transferability of unlearnable examples (UEs) across different learning paradigms, i.e., supervised learning and self-supervised contrastive learning. Different from existing works including both supervised UE generation methods and hybrid methods, this paper argues that strong data augmentations with supervised learning are sufficient for unsupervised unlearnability. It then proposes two strong data augmentations equipped UE generation methods AUE (with standard training) and AAP (with adversarial training). The two proposed methods demonstrate certain superiority over the existing methods on low- and high-resolution datasets.
strength_weakness: strength_weakness: 1. The alignment and uniformity metrics are all from existing works, which lowers the novelty of the finding in Table 1: "contrastive unlearnability is highly related to huge alignment and uniformity gaps". And the two proposed methods have little to do with the two gaps?

2. While the finding that "supervised learning with strong data augmentation can achieve unsupervised unlearnability" is interesting, the two proposed methods have limited technical novelty. In other words, they do not explore the power of data augmentation more systematically. The current mechanism is very ad-hoc.

3. The performance difference (AUE vs AAP) between low- vs high-resolution datasets is intriguing. AAP targets adversarial training exploitation, so why it should perform better or worse than AUE against contrastive learning? Does contrastive learning have anything to do with adversarial training?
questions: questions: 1. Should the worst-case unlearnability contain other learning paradigms rather than just supervised learning and contrastive learning? For example, MAE or autoregressive pretraining. We cannot enumerate all training paradigms.

2. Why data augmentation is so important for obtaining unsupervised unlearnability, what if there are non-data-augmention-based unsupervised learning methods?

--------
Post rebuttal: I have increased my rating according to the authors' responses.
limitations: limitations: The technical novelty of this work is somewhat limited.

Comments: title: Discussion about DiffAug
Further clarification
Thanks for the clarification
Thank you for your detailed response
comment: I appreciate the authors' response. My concerns regarding the theoretical analysis and technical contributions have been mostly addressed. I would like to raise my score accordingly.
Dear Reviewer J4Hn,

We have submitted a rebuttal addressing your comments. Could you please review it and confirm if your concerns have been resolved?

Best regards,\
Authors
Thank you for sharing the recent papers. We have not tested them. In the future version, we will discuss them in more detail in the relevant sections.
Thank you for your detailed response and the effort in testing against DiffAug. Did you test the attack performance against more typical diffusion purification approaches like those SDEdit-based or super-resolution-based [1,2,3]? 

[1]. Diffusion Models for Adversarial Purification, ICML'22
[2]. Black-box Backdoor Defense via Zero-shot Image Purification, NeurIPS'23
[3]. Pixel is a Barrier: Diffusion Models Are More Adversarially Robust Than We Think
Thank you for acknowledging our work.
We will address your concern about DiffAug through additional experimental results.

Since DiffAug's public repository only contains code for biology datasets, we sent emails to the authors requesting official implementations for vision tasks. Unfortunately, we have not received a response so far. 
Consequently, based on the paper and the existing code, we did our best to reproduce DiffAug on CIFAR-10. We used a ResNet-18 as the encoder backbone and a UNet as the diffusion backbone. 

We train the DiffAug on clean/poisoned training data and then perform linear probing upon the encoder. The following table shows the test accuracy. **Our attacks successfully transfer to DiffAug**. Specifically, compared to the non-attack case, AUE and AAP attacks reduce the test accuracy by 46.44% and 50.48% respectively.

| Clean | AUE | AAP |
|-------|-----|-----|
| 81.88 |35.44|31.40|

In summary, although DiffAug incorporates diffusion-based data augmentation in contrastive learning, our proposed attacks are still effective against it.
Thank you for your further explanations that addressed my concerns. I have adjusted my recommendation.
Thanks for your response to our rebuttal! We will further clarify our method and the motivation of it in detail. 

**[1] Motivation of augmentation enhancement**
To make clear our motivation, we first revisit the mechanism of error-minimization and then clarify the motivation and the statement in Line 209.

- **Mechanism of error-minimization** In Equations (2,4) of our paper, error-minimization refers to a min-min optimization of an objective (SL or CL) loss with respect to weights and perturbations. It aims to generate perturbed data that can be extremely easily optimized by the cross-entropy loss for SL or by InfoNCE loss for CL. 

    As a result, this characteristic of poisoned data, which makes the loss function converge easily, leads to a model that only learns the fraudulent synthetic patterns from perturbations and fails to learn the true ground-truth data distribution during training. For example, in Figure 7 of our paper, we closely examine the training process on poisoned data and find it converging rapidly.

- **Motivation of our method and the statement in Line 209** 
To obtain unlearnability for CL, the existing path is contrastive error-minimization (CP, TUE, TP) based on the CL training process. 
In this paper, we aim to provide a solution based on the SL training process which is more efficient. 

    Recall that Equation (2) differs from Equation (4) with the loss function choice, say, SL loss vs. CL loss. Our **key insight** is, that using contrastive augmentation in SL loss and optimizing it can implicitly reduce the CL loss to some extent, as shown in Figure 4. (More empirical observations and theoretical analysis about this have been discussed in Section 4.1 and Appendix E.)

    Thus, for our proposed AUE attack, while it appears to be performing supervised error minimization, it is also carrying out contrastive error minimization.
    That is, as stated in **Line 209**, the enhanced data augmentation allows supervised error-minimization to partially serve the role of contrastive error-minimization. We verify this claim in Figure 5(a) of our paper, in which the InfoNCE loss during training on AUE poisoned data is largely reduced, achieving a similar effect of contrastive error-minimization.

**[2] Implementation schemes of augmentation in our attacks**

- **Augmentations in SL and CL** CL typically uses more and stronger data augmentations compared to SL. Specifically, SL generally relies on simpler augmentations like cropping and flipping, whereas contrastive learning incorporates more advanced techniques such as color jittering, grayscale, and others.
SL-based attacks typically leverage the mild data augmentation used in supervised learning.


- **Default constant strength scheme** 
To enhance the contrastive unlearnability of SL-based attacks, we replace the data augmentation with contrastive augmentation and adjust the intensity of augmentation through a strength parameter, as shown in Appendix C.2, Code Listing 1.
By default, we fix the augmentation strength as a constant value in the generation of AUE and AAP attacks, as discussed in Appendix D.4.
Comprehensive experiments show that our attacks are both effective and efficient against SL and CL simultaneously.



- **Annealing and tempering schemes** 
In the previous rebuttal, besides the constant augmentation scheme, we also try annealing and tempering schemes. These two dynamic choices are inspired by [1], which proposes that annealing down augmentation strength is beneficial for adversarial contrastive learning. We want to see if dynamic schemes also benefit our problem.

    As shown in Figure 1 of our rebuttal, on CIFAR-10, the improvement of a particularly selected alternative scheme is marginal compared to the default one. Therefore, we believe that the **decisive factor** in our method is the contrastive augmentation itself, rather than the choice between constant and dynamic schemes for the augmentation.

[1] Luo R, Wang Y, Wang Y. Rethinking the effect of data augmentation in adversarial contrastive learning. ICLR 2023.


We hope our reply makes things more clear. Feel free to let us know if you still have concerns.
Thank the authors for your detailed clarifications and additional analysis. I appreciate all of your efforts. I still have one concern regarding the "enhanced augmentations". In Figure 1 (b) of the rebuttal, does it suggest that both annealing and tempering are not critical for the unlearnable CL? Could you please further motivate your choices? To make it more specific, in line209 of the draft: "In other words, supervised error-minimizing noises with enhanced data augmentations can partially replace the functionality of contrastive error-minimizing noises to deceive contrastive learning." Could you further clarify the enhanced augmentation and motivate it?
Thanks. Most of my concerns have been addressed, thus I increase my rating to 6.
Thank you for your detailed response. Your response addresses most of my concerns. I will raise my score. But I am still looking forward to seeing the perturbation against DiffAug experiments, since, as you mention, it's a fundamental challenge in this field.