Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: Great Start (GS): a submission which meets some of the reviewing criteria but has room for improvement
review_content: summary_of_contributions: The paper presents a new dataset of individuals identifying as of Indian ethnicity for face attribute identification. It also presents the results of the evaluation of a soft biometrics predictor using the proposed dataset.
strengths_and_weaknesses: The paper presents an analysis on the performance of an existing soft biometrics recognition network when tested in the wild on people identifying as of Indian ethnicity. The analysis reviews the known and important issue of bias towards the data distribution seen during training, that can heavily affect performance when tested on different data.  
To do so, a new video dataset is collected using mobile phones and fixed acquisition distance in unconstrained environments.  
While this paper presents only an analysis of existing methods, the authors mention that future work will include a novel, de-biased gender classification and age estimation networks.

The paper also presents some weaknesses.
- Some important details of the proposed dataset are missing, such as the number of M/F subjects and their age distribution, the video and image resolution, and whether the dataset is expected to be published.
- Four different face detection methods are tested, while only one soft biometrics predictor is evaluated. Given that the focus of the paper is on soft biometrics recognition performance, I would have expected the evaluation of different recognition methods rather than different face detection models.
- According to Serengil & Ozpinar (2020), their model expects the face to be detected by OpenCV and horizontally aligned, but I didn't find this alignment step mentioned in the text.
- The Table in Figure 2(a) is wrongly formatted (MTCNN and OpenCV span one column instead of 2) and it could contain both relative and absolute values.
- The English level is good in general, but the text contains some minor typos (e.g. suscept instead of suspect)
- Overall, I'm not totally convinced by the motivations provided in the abstract and the introduction. They seem to be a specific use case (surveillance/policing), which also raises ethical questions, while soft biometrics can be used in a variety of different tasks.

I also wonder if the degraded performance of LightFace is caused by the ethnicity of the participants or by the very low resolution of their face. As future work, the authors could consider to extend their dataset with recordings at different distances to assess how much the face resolution affects the overall performance.
suggested_changes: I suggest to include additional information about the proposed dataset in the text (an Appendix section can be added if needed), proofread the text for typos, and improve the formatting of the Table in Figure 2(a).

I also encourage the authors to add an "Ethics Statement" section, as recommended in the [Author Guide](https://iclr.cc/Conferences/2023/AuthorGuide), to discuss the adherence of their paper to the [ICLR Code of Ethics](https://iclr.cc/public/CodeOfEthics).
In particular, I think that two points should be discussed: the collection of the dataset and the publication approval of the participants, and the potential consequences that effective in-the-wild soft biometrics predictors could have on the society if used for surveillance/policing.  
The ethic statement does not count toward the page limit, but should not be more than 1 page.
review_title: null

final_score: rating: Needs Clarification (NC): a submission which does not meet the reviewing criteria and needs clarification for its described problem or solution
review_content: summary_of_contributions: This paper proposes a new facial dataset with unconstrained images taken in the real world. Following that, it also conducts experimental analyses to determine whether current facial recognition algorithms are fair and unbiased for the tasks of age and gender prediction.
strengths_and_weaknesses: **Strengths:**

- The authors test multiple different facial recognition backbones as part of their experiments.

**Weaknesses:**

- (**Note to PCs**) I have concerns about the ethical considerations of this work. The authors have proposed and collected data, including facial images, for this paper. Unfortunately, there is no clarification provided on whether consent was obtained from the subjects.
  - This is further exacerbated, given that the data is being used to train models to estimate the gender and age of the subjects.

- The authors issue a blanket statement denouncing most facial recognition models as biased (especially towards Indians). However, they fail to make any comparisons with models trained on UTKFace [1] or FairFace [2], datasets that are known to incorporate different races and identities. In fact, the authors fail to show any quantitative results that back their claims of models being biased.

- There also seem to be no experiments with current state of the art models in gender and age estimation.

````
[1] Krishnan, A., Almadan, A., & Rattani, A. (2020, December). Understanding fairness of gender classification algorithms across gender-race groups. In 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA) (pp. 1028-1035). IEEE.
[2] Kärkkäinen, K., & Joo, J. (2019). Fairface : Face attribute dataset for balanced race, gender, and age. arXiv preprint arXiv:1908.04913.
````
suggested_changes: - Leaving ethical concerns aside, this paper deals with complex social issues pertaining to the invasion of privacy and data usage. The wording of the paper (especially for identifying people in public places) leaves a lot to be desired. At the very least, I would suggest adding some mention and add references to how bad actors can misuse these technologies for personal gain.

- Based on reading the paper, the authors have mentioned the data being collected with a handheld mobile camera recording people in the real world in unconstrained settings. I sincerely hope appropriate permissions were sought before collecting data and making predictions.

- The experiments in the paper show that the backbones finetuned on the dataset provided by the authors are ineffective for age/gender estimation. However, there exist subtle differences between generic and fine-grained image analysis. The backbone networks (part of the DeepFace library) used by the authors pertain to the latter, while the dataset used by the authors is closer to the former. Appropriate measures (like using different models or backbones or changing the architecture entirely) should be used for a fairer comparison.
  - The authors have mentioned developing a better, de-biased model, which may be a step in this direction.
review_title: null

final_score: rating: Great Start (GS): a submission which meets some of the reviewing criteria but has room for improvement
review_content: summary_of_contributions: TL;DR: The paper investigates face recognition and attribute analysis of Indian individuals captured in uncontrolled settings using mobile devices. The authors find that current deep face recognition models are ineffective in age estimation and demonstrate gender bias towards males, with poor performance in the classification of females. They also highlight that the success of face recognition algorithms in semi-constrained datasets may not necessarily translate to real-world scenarios.
strengths_and_weaknesses: ## Strengths:

1. The authors curated a real-world dataset for Indian ethnicity individuals in unconstrained settings at a distance.
2. The paper presents a detailed experimental analysis of deep face recognition algorithms and highlights the need for unbiased models for accurate face attribute analysis.

## Weakness:

1. Though the paper emphasizes face as a nonintrusive modality, it is considered intrusive. According to GDPR, Using biometric data such as facial images is very restricted under the GDPR (Article 9).
2. Though the paper claims, there is a lack of images from Indian ethnicity, it can be found in the UTKFace dataset and gender information in the Adience dataset.
3. Please consider evaluating works by [1] and [2] in your future study.
4. A sample size of 30 is a bit limiting to conducting the study. Please consider increasing the sample size and using other available datasets such as UTKFace and the Adience dataset.

## References:

1. Abdolrashidi, Amirali, Mehdi Minaei, Elham Azimi, and Shervin Minaee. "Age and gender prediction from face images using attentional convolutional network." arXiv preprint arXiv:2010.03791 (2020).
2. Levi, Gil, and Tal Hassner. "Age and gender classification using convolutional neural networks." In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 34-42. 2015.

suggested_changes: 1.  please correct a possible typo  suscept->suspect
2.  A sample size of 30 is a bit limiting to conducting the study. Please consider increasing the sample size and using other available datasets such as UTKFace and the Adience dataset.
review_title: null

Meta-review: metareview: The paper presents a new dataset of individuals identifying as of Indian ethnicity for face attribute identification, and evaluates a soft biometrics predictor using the proposed dataset. The authors analyze the performance of an existing soft biometrics recognition network when tested in the wild on people identifying as of Indian ethnicity. The analysis reviews the known and important issue of bias towards the data distribution seen during training, which can heavily affect performance when tested on different data. The paper presents a detailed experimental analysis of deep face recognition algorithms and highlights the need for unbiased models for accurate face attribute analysis.

Review 1 notes that the paper lacks important details of the proposed dataset and suggests the evaluation of different recognition methods rather than different face detection models. The reviewer also questions the motivations provided in the abstract and the introduction and mentions the degraded performance of LightFace, which may be caused by the ethnicity of the participants or the low resolution of their faces. As future work, the authors could extend their dataset with recordings at different distances to assess how much the face resolution affects the overall performance.

Review 2 raises concerns about the ethical considerations of the work and suggests that there is no clarification provided on whether consent was obtained from the subjects. The reviewer also notes that the authors fail to make any comparisons with models trained on UTKFace or FairFace datasets that incorporate different races and identities. Moreover, there seem to be no experiments with current state-of-the-art models in gender and age estimation.

Review 3 notes that the authors curated a real-world dataset for Indian ethnicity individuals in unconstrained settings and presents a detailed experimental analysis of deep face recognition algorithms. The reviewer suggests that using biometric data such as facial images is restricted under GDPR, and there are other datasets available that include Indian ethnicity, such as the UTKFace dataset and the Adience dataset. The reviewer suggests increasing the sample size and using other available datasets to conduct the study.

Overall, the paper's main contribution is the proposal of a new dataset and the evaluation of a soft biometrics predictor using the proposed dataset. The paper highlights the importance of unbiased models for accurate face attribute analysis and presents a detailed experimental analysis of deep face recognition algorithms. However, the paper lacks important details of the proposed dataset, and there are ethical concerns regarding the collection of data. Moreover, the authors fail to make any comparisons with models trained on other datasets that incorporate different races and identities.

confidence: 4: The area chair is confident but not absolutely certain
Decision: Invite to revise