Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper describes a method for making user data unusable for training machine learning models. It focuses primarily on image data. The basic idea is to use error-minimizing noise. 

In this paper the author propose adding imperceptible to users error-minimizing noise that would make training data unusable for training. The authors proposed 2 methods for generating the noise: sample-wise and class-wise

This paper is well written. The code and the datasets used for the experimentation have been provided. 

########################
Overall, I would recommend accepting this papers. My only concern is with the effectiveness of the proposed technique given what authors discussed in the appendix (see questions below).

The method was used on standard openly available image datasets. The results showed that when close to 100% of training samples have been updated with the error-minimizing noise the model performance went down considerably (as desired). However, when even 20% of training data was left clean model performance remained good. 

########################
Questions:

From the appendix notes: it appears that adversarial training can significantly negate the effect of adding error-minimizing noise. The resulted model performance would be degraded when compared to model training using clean training data only but considering the fact that authors themselves acknowledged that user data with error-minimizing noise may just constitute a fraction of all the training data available for training a model the effectiveness of this technique may be limited.  (due to the effectiveness of adversarial training and the outsized influence of a relatively small number of clean data samples on model performance)
Can the authors discuss the issues with the effectiveness of their presented technique?


Mostly cosmetic:
page 5:  section 4.1 title “Error-maximizing” written twice.
review_title: Unlearnable Examples: Making Personal Data Unexploitable.  

final_score: rating: 7: Good paper, accept
review_content: Summary:
The authors studied the problem of data protection from a new perspective. They proposed one kind of error-minimizing noise to make the data (added noise) unlearnable. The noise is imperceptible to human eyes, and thus does not affect normal data utility. The idea is very interesting and inspiring. The authors conducted a series of solid experiments to validate the effectiveness of the proposed noise, and tested it on a real world task of face recognition. 

Pros:
1. The idea of the paper is very interesting. Its motivation is intuitive and well explained. Considering adversarial training is to find the worst case example to make the training process robust, the authors proposed an opposite direction to find the easiest case to make the training process to learn nothing. The authors also proposed two types of noise: class-wise and sample-wise, which is a complete formulation. 
2. The paper revealed an important problem to protect privacy, and proposed a simple yet effective method to prevent our data from unauthorized exploitation for training commercial models. I think it will attract a broad audience in the ICLR community.  
3. The experiments are solid and comprehensive, considering the difference to random and error maximizing noises, effectiveness on different datasets and model architectures. The detailed stability and transferability analysis convince me why and how error minimizing noise works. Besides, they also show a real-world face recognition task to demonstrate its usefulness in practice.  

Cons:
1. What is the overhead of generating and adding this kind of noise? The author did not mention it in the paper. 
2. Revisiting Figure 1, I am curious to know why the sample-wise and class-wise noises perform so differently? especially for random and error-maximizing noise?
3. What is the difference between the proposed noise and the data poisoning methods?
review_title: Review 2 for "Unlearnable Examples: Making Personal Data Unexploitable " 

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary:
The authors proposed the idea of using invisible noise to make personal data unusable to authorized deep learning models. To achieve this goal, the authors proposed the idea of error-minimizing noise crafted by a min-min optimization method. The error-minimizing noise is then added to training examples to make them unlearnable to deep learning models. The idea is very well motivated and explained. The experiments not only confirm the exceptional effectiveness of the proposed method but also show its flexibility.

Pros:
1. The paper is very well written and easy to read.

2. I find the idea is very attractive and could have a significant social impact, especially considering the fact that personal data has already been overused without consent to train not just commercial but also illegitimate models to fake information or track people’s identity. 

3. The idea of using the error-minimizing noise is well explained and the generation method is well formulated.

4. The experiments are very thorough, providing not only evidence of the superb effectiveness of the proposed noise over random or adversarial noise, but also the flexibilities and limitations of the proposed method. The real-world experiment makes the proposed idea even more convincing, although it is just a simple simulated scenario.

5. It seems that class-wise noise can easily break a classification model, which is somewhat interesting from the data protection perspective.

Cons:
1. I think the class-wise noise breaks the IID assumption of machine learning. It seems that breaking the essential assumptions in machine learning can break the model. Although this is not new, however, it turns out to be very interesting if used for data protection or similar ideas. The authors could have more discussions on this point. For example, what would happen if someone always used a different background (may be invisible) for each of the photos uploaded to social media, always shifting the newly collected test data to a different distribution. Can this serve as the same purpose?

2. The proposed noise seems not strong against adversarial training. Although adversarial training is costly and decreasing performance at this moment, they may be improved in the future. A discussion on the possible ways to generate the noise against adversarial training can be useful. 

3. How the proposed method is related to backdoor attacks? It acts as a type of backdoor attack. Yes, backdoor attacks do not decrease the model’s performance on clean data. I think the “clean data” in the proposed setting should be the “poisoned data” rather than the ground truth clean data since both the training and testing data will be collected at the same time. I guess the only difference is that, in this protection setting, the defender cannot do anything about it unless recollecting or denoising the data, even if the defender finds the model is poisoned. I suggest the authors include more discussions around this point.
review_title: The idea presented in this paper is very interesting

final_score: rating: 7: Good paper, accept
review_content: 
$\textbf{Comments:}$
The paper's motivation is based on protecting private data and preventing its being scraped and used to train models. Even though motivation is clear and very important, the problem is the same as the works in crafting adversarial samples (i.e., the ones under data poisoning and adversarial attacks parts of the related work). The key difference is to apply Projected Gradient Descent (Mandry et al. 2018) in the reverse direction iteratively to *minimize* the loss function.  Furthermore, the performance evaluation will be the margin between models trained on completely clean data and sample-wise/class-wise adversarially corrupted data (in contrast to fooling a pretrained network in adversarial attack benchmarks). 

$\bullet$ *Percentage of noisy training data:* In the "Assumptions on Defender's Capability" paragraph, the assumption is that only a part of the training set could be perturbed. The margin between error maximization and minimization on CIFAR-10 is remarkable (Figure 1), and this figure is misleading. 100\% of the training data was perturbed. 

Besides, Table 2 gives accuracy in different ratios of noisy training samples. To understand whether perturbed training samples contribute to learning or not, I would compare them with clean training. For instance, in addition to the results of 20\% perturbed training setting in $\Delta_s$ and $\Delta_c$, training with only 80% of clean data without perturbed samples. 

$\bullet$ *Comparison to PGD (Mandry et al. 2018):* Even under the class-wise perturbation, the noisy training data is learnable. In a sample-wise setting, "error-maximizing noise" is still learnable and performs very well; however, it performs around 20 and similar to "error-minimization" in a class-wise setting (Figure 1). If I am not wrong, Projected Gradient Descent, as proposed and applied in Mandry et al. 2018 (Figure 1, right side), reduces the performance the same as the proposed error minimization approach, and there is no performance gain.

$\bullet$ *Generalization to different Adversarial Attack methods:* Error minimization is shown using PDG only. There are several adversarial attack benchmarks on CIFAR-10 and ImageNet, such as CleverHans, Foolbox, or Realsafe (Considering different evaluation protocols, adopting these benchmarks for evaluation is a reasonable option to eliminate other factors). Is error minimization limited to only PDG or other methods? Did you try the effect of error minimization using any other method?

https://arxiv.org/abs/1707.04131

https://arxiv.org/abs/1610.00768

https://arxiv.org/abs/1912.11852

$\bullet$ *Different source-target models:* In all experiments, the source model is Resnet-18. The classification models used in performance evaluation are ResNet-18, ResNet-50 and DenseNet-121. All three models are based on residual blocks. In practice, we cannot assume the architecture that will be used by third-parties. Did you try completely different target models (such as AlexNet VGG,  Inceptionv3, etc.) 

$\bullet$ *Application to face analysis:* Face recognition experiment is non-standard. I strongly recommend applying a standard dataset evaluation that would make comparisons possible. Both source and target sets are the datasets' subsets, and the selected identities might show visual (dis)similarities (i.e., ethnicity, age, gender). You can report the full performance on the entire target dataset (WebFace). Furthermore, face recognition models are trained as a recognition problem (with classification losses or metric learning) but tested in face verification settings (calculating the distance to query samples).  Reporting the distribution of these distances, for instance, Cumulative Matching Characteristic (CMC) and Receiver Operating Characteristic (ROC), would be more informative.
review_title: This paper proposes a method to make personal data unexploitable. The objective is to introduce imperceptible noise to make the data robust against being automatically scraped and used to train commercial models. The idea is to make small sample-wise or class-wise perturbation in the direction that minimizes error and eventually makes the models trained on this model believe that there is nothing to learn from these samples.

Meta-review: The paper proposed a *novel* methodology for protecting personal data from unauthorized exploitation for training commercial models. The proposal is conceptually *intuitive* and technically *motivated*. It goes to the opposite direction of adversarial training: by adding certain error-minimizing noise (rather than error-maximizing noise) to the data, the model is fooled and believes there is nothing to learn from the data, and thus this can protect the data from being used for training. The paper is of not only *high quality* but also *broad interest* given the current social concerns about personal data privacy. I think its potential impact should get it a spotlight presentation.
Decision: Accept (Spotlight)