Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents an empirical study on the effect of face obfuscation in the ImageNet dataset. The main conclusion is that face obfuscation does not decrease the utility of the dataset. Specifically, the authors showed that various networks trained on the obfuscated dataset only experienced small accuracy drop on the image classification task. The authors also discussed the impact on different categories, showing that face obfuscation hurt more to the object categories that are more closely related to faces (i.e., the bounding boxes of which overlap more with faces). Last but now least, experiments has been conducted to show that face obfuscation also does not have a significant impact on the transferability of the features learned from the new dataset. All these conclusions are inline with intuitions since ImageNet is not primarily focused on human activities / faces.
main_review: Strengths:
* The main contribution of this work is that it provided empirical evidence on the effect of face obfuscation on the ImageNet dataset. Through comprehensive experiment, the authors showed that face obfuscation does not decrease the utility of the dataset.
* Another contribution that should not be overlooked is that the authors annotated all faces in ImageNet in a semi-automatic manner and promised that they will make the annotations publicly available to other researchers.
* The paper is very well written and includes a lot of details on the experiment protocol. Thus It should be straightforward for other researchers to reproduce the results and to extend the study.

Weaknesses:
* Since blur and cutout are commonly used data augmentation techniques, it is to be expected that face obfuscation would not have a big impact to visions tasks that have little to do with faces. Although it is commendable that this is now shown empirically though the study, this work also does not bring interesting new insights into the topic.
* The authors showed that categories that are closely related to faces are indeed affected more by face obfuscation. This paper would be more interesting (from a technical point of view) if the authors could additionally investigate into methods for alleviating such impact.
summary_of_the_review: This paper is very well written and it provides empirical evidences to support the intuition that face obfuscation does not decrease the utility of the ImageNet dataset. However, my main concern is that the paper has no technical novelty and it also does not bring sufficiently new insights to the community.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper mainly discuss the privacy issue for human for the widely used ImageNet dataset and how to handle them.
In addition, the paper does a very detailed empirical experiments to study the performance influence for various tasks, including object recognition, scene recognition, face attribute, and object detection if all the faces in the ImageNet are obfuscated.
main_review: The main strength of the paper is to address the privacy issues of ImageNet and is to provide an alternative face obfuscated version.
In addition, the authors conduct very thorough experiments across different tasks and different architecture for the study of the performance influence with the obfuscated dataset. It shows new dataset still is effective for transfer learning for various vision task with few performance drop. However, the weakness is that the main part of the paper is to examine the performance influence of different settings and is of limited technical novelty. For the verification of some downstream tasks, the coverage of tasks is not enough.
summary_of_the_review: Although the paper mainly focuses on providing plenty of empirical results to evaluate the influence of using the face obfuscated ImageNet dataset and is of limited novelty, it provides a lot of insights to show the effectiveness and feasibility of privacy preserving ImageNet.

I have few concerns of the selection of transferring tasks. For example, the resolution of CIFAR-10 is only 32x32 and only for 10 classes. Similarly, Pascal VOC is also relatively small and easy dataset as compared with COCO or other recently released object detection dataset.
Most of the images in the CelebA dataset are in frontal pose and have much fewer variations than other unconstrained face dataset, like IJB-C, etc. Since these datasets are relatively simpler than others which are more close to real-world scenarios, I wonder if the same experimental results and findings are still valid for harder datasets with more variations.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The main concern addressed in this paper is the privacy problem that may result from images in ImageNet databases containing unexpected faces. The authors propose a two-step face filtering method. First, the authors use a detector called Amazon Rekognition to detect the ImageNet database. Then, the authors further optimize the detector output through the crowdsourcing platform Amazon Mechanical Turk (AMT) to reduce false positives and false negatives in automated detection. For the detected faces, the authors took two approaches, distinguishing between blurring and overlaying, and tested their effectiveness on different models separately. The accuracy of the two approaches was reduced by 0.9% on average compared to the original database on the ILSVRC classification challenge. And using the database that has blurred or covered the faces still maintains the transferability of the original database in the tests of downstream tasks.
Contribution：
1. The authors perform a very time-consuming and labor-intensive task for accurate labeling and filtering of faces in the ImageNet database and statistical analysis of the classes of faces contained in ImageNet.
2. The authors demonstrate experiments related to classification tasks and pre-trained model training using a database containing blurred or covered faces, proving that the theory is feasible and that the dropped accuracy is acceptable.
3. In terms of ethics, using blurred or covered face data for training can reduce privacy concerns. The study of the ImageNet database in terms of privacy can provide an important reference for subsequent databases
main_review: The main weakness:
1. I was confused by the statement in section (Part III, second paragraph) as to whether the face annotation process was manually filtered only for faces that were successfully detected (generated prediction boxes5) by Amazon Rekognition. Although I can infer from Appendix A Stage1 that only the data with successful detector predictions should be put into the AMT platform. Perhaps it would be better to include a brief description of Amazon Rekognition in the text. 
2. And if some faces are not detected by Amazon Rekognition, how do you tackle this problem? This is an important problem for the privacy issue in this paper because this paper focuses on the privacy issue.
3. The novelty of this paper is limited reference to the proposed method in the paper.

The main strengths:
1. The experimental part of the paper is depicted in great detail and completely. Rigorous validation of the obfuscation approach is done on two different methods on 15 different models. And in the appendix, the detailed method of blurring, and the problems that may happen in the process of labeling are under clearer explanation.
2. The ethics of machine learning has been widely debated, with the issue of face privacy being of particular concern. There are many similar discussions, for example, there are some papers proposing to remove images associated with people from the database. The feasibility of obfuscation processing of faces proposed in this paper is a good way to minimize privacy issues without reducing the number of databases at the same time.
summary_of_the_review: This paper has some contributions in exploring the ethicality of datasets, especially in the current very popular ImageNet database, but it exists some flaws (see weakness). The solutions and results in this paper are open sources and feasible, and this work will inspire subsequent exploration of privacy protection in publicly available datasets.
review_title: null

Meta-review: This paper received 3 quality reviews, with 2 rated 5 and 1 rated 6. While the reviewers recognize the various contributions and insights made by this work, it was also pointed out that this work lacks technical novelty. The authors agreed with this concerns and argued that this work provides a service to the community, citing imageNet and COCO papers. The AC agrees with the contribution and major concerns. Furthermore, the AC would like to point out that in term of the level of efforts, this work might not be on par with the imageNet and COCO. All things considered, the AC believes that this work is not ready for publication at its current form, and hence recommend rejection.
Decision: Reject